From 1ed7f0ddf243dd94aea2febbcdf1e91dc6de2f05 Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 19 Oct 2026 02:51:32 -0700 Subject: [PATCH 01/12] docs(agents): compress Key Lessons to one law per bullet, correct the FA-quants law MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Key Lessons Learned 483 -> 340 lines. Every law survives; only the essays went. Port-before-cut census over the diff: 454 deleted lines, 0 orphaned facts (every number still has a holder in the vault). - Each lesson is now one bullet with the page that owns the detail. - De-duplicated the page's own internal repeats: the layer-split law was told twice (PCIe and ROCm sections), the FA-quants law twice more. - CORRECTION, verified on dankclaw 2026-10-19: `GGML_CUDA_FA_ALL_QUANTS` is deprecated upstream (master's ggml/cmake/common.cmake tells you to use GGML_CUDA_FA_QUANTS=all), the successor's default pair list `q4_0-q4_0;q8_0-q8_0;f16-f16;bf16-bf16` compiles the quantized-KV kernels with the old flag OFF (master build-cuda cache: FA_ALL_QUANTS=OFF), and neither llama tree references the old macro under ggml/src/ggml-cuda/. The "required or prefill collapses ~12x" law is obsolete; the new law is to read the CMakeCache pair list. - frank-zealot: ported the three facts AGENTS.md alone carried, and verified them against the host first — the 2026-10-18 13:16 cutover boot really did load PTQ1_0 at `new slot, n_ctx = 133888` x3 (journal), the mmproj fit check really printed 873.10 MiB against a 21,439 MiB projection, and the warmup spread is 2.6 -> 1,258 t/s prefill. - New tooling: scripts/check-okf.py (OKF v0.1 conformance linter) and scripts/gen-vault-index.py (root index regenerated from frontmatter). Co-Authored-By: omp --- AGENTS.md | 765 +++++++++------------- scripts/check-okf.py | 175 +++++ scripts/gen-vault-index.py | 135 ++++ serverAppConfigs-docs/LLM/frank-zealot.md | 15 +- 4 files changed, 631 insertions(+), 459 deletions(-) create mode 100755 scripts/check-okf.py create mode 100755 scripts/gen-vault-index.py diff --git a/AGENTS.md b/AGENTS.md index e5c817b..b41f980 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -475,481 +475,338 @@ linked runbooks. **PCIe / multi-GPU** -- **RCCL collectives need PCIe atomics on every participating GPU.** RCCL's - kernels carry a hidden hostcall-buffer parameter; ROCr refuses to dispatch - them when the device has no atomics, and the failure surfaces as - `hipErrorIllegalState` ("the operation cannot be performed in the present - state") from `hipLaunchKernel`, not as anything that names atomics. The - value is consumed at driver probe time, so writing the AtomicOp Requester - bit with `setpci` at runtime changes nothing. Check the *chain*: - `lspci -vv -s ` and every upstream bridge must show - `AtomicOpsCap: Routing+` (a `Routing-` bridge is a hardware wall), and the - endpoint's Device Control 2 (`setpci -s CAP_EXP+28.w`) must have bit - 6 set. AMD desktop platforms put one card on CPU root ports (atomics work) - and the rest behind the chipset (usually not) — that asymmetry is what - blocked frank-medium's TP=2. A one-card collective proves nothing: NCCL - short-circuits at `world_size=1`. -- **That wall was physical, and it was fixed on 2026-09-20 by a board swap.** - Both paths that put two cards on CPU root ports were: an M.2→PCIe x4 riser - into a *CPU-attached* M.2 slot, or a board whose CPU lanes bifurcate to - x8/x8. dankserver took the second: **ASUS ProArt B850-Creator WiFi NEO**, - `PCIEX16_1` + `PCIEX16_2` at x8/x8 Gen5 on root ports `00:01.1` / `00:01.3`, - every bridge now `Routing+` and **both endpoints `ReqEn+`** — the exact - precondition the old test plan named. Bandwidth was never the constraint (a - 27B TP=2 allreduce at batch 4 is ≈10 MB per token-step, ≈1.4 ms at Gen4 x4 - and under 0.4 ms at Gen5 x8). - **Proven on the same day.** vLLM's R4D one-shot allreduce runs TP=2 here and - frank-medium serves on it; llama.cpp's RCCL path was built too - (`GGML_HIP_RCCL=ON`) and its tensor mode measured — real collective, slower - shape. Note the collective choice matters: TheRock's RCCL builds lack gfx1201 - device code (`invalid device function`), which is why R4D is the path vLLM - uses at all. Evidence matrix and the measured three-arm verdict: - [[Dual-R9700-Cooperation]]. -- **When a GPU collective dies, read the runtime, not the framework.** The - message that identified the cause came from `AMD_LOG_LEVEL=3` - (`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch - failed!`), not from vLLM or RCCL. `AMD_LOG_LEVEL` costs one env var and - prints the failing HIP call with its return code. -- **The software path that puts both cards on ONE model without a collective: - llama.cpp's layer split.** On the old board vLLM could not do it in any of its - parallelisms — TP needs an allreduce, PP needs the same p2p primitive, and both - died on the card that could not dispatch an RCCL kernel. `--split-mode - layer` in llama.cpp needs no - collective: layers are distributed in order and only the boundary activation - crosses PCIe, once per forward pass. Measured here: 1991 t/s prefill / - 18.8 tok/s decode layer-split, against 272 / 26.3 tensor (butterfly - fallback). - This was what `frank-medium` ran until 2026-09-20, when the board swap made - collectives legal and radiance TP=2 beat it on every axis (95.8 vs 45.7 t/s - decode, 0.21 vs 0.72 s TTFT, 259 vs 72 t/s burst-4) — see - [[frank-medium-radiance-mxfp4]]; the llama shape is the rollback target - [[frank-medium-llamacpp-dflash2]]. -- **Layer split is also the cross-vendor answer — and `-ot` is what cannot be - combined with it.** On Dankclaw (2026-09-16) a CUDA 3090 + Vulkan iGPU layer - split ran a real two-device graph: `-dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85` - held 14,994 MiB on the 3090 and gave 27.2 tok/s with 268.5 t/s prefill at 64k - depth (iGPU alone: 25.8 / 231.8). What does NOT work is per-tensor placement - across devices: `--override-tensor =` sets - `has_tensor_overrides()`, a REQUIRED term of the scheduler's - `pipeline_parallel` flag (llama-context.cpp:433 — the only place in the tree - that reads it), and that flag is what sets `n_copies` in - `ggml_backend_sched_new`; with `n_copies == 1` a pre-allocated weight on a - non-primary device has no backend and the graph aborts (`ggml-backend.cpp:941`, - reported with the leaf's own op name `(NONE)`). `=CPU` is the only override - target that works, because the CPU backend is last in the list. Pure Vulkan - changes nothing here — the gate is the override, not the driver mix. See - [[x4-pair-experiment]]. -- **A host↔3090 link at x4 measures 6.72 GB/s, and that is a design - constraint, not a detail.** `pcie-link-test.cu` on Dankclaw (2026-09-16, - services stopped): pinned H2D 6.72 GB/s, D2H 6.60 GB/s — 85 % of the Gen4 x4 - ceiling — with async no better than sync and pageable within 1 % of pinned, - i.e. the copies are link-limited rather than staging-limited (the usual - "pageable costs 2x" rule does not apply here). The card is x16-capable and the - Strix Halo port beneath it is x8-capable (`LnkCap: 16GT/s, Width x8` on - `00:03.1`), so the x4 lives in the adapter or cable; the link idles at - 2.5 GT/s x4 and trains up under load. Consequence for any engine that - *streams* weights across it: a 3 MiB slab costs 474 us, and a 111 GiB A3B - model needs ~1.70 GiB of active experts per token ⇒ **3.7 tok/s** (4.6 with a - 20 % resident cache), against **20.46 tok/s** for the same model with the - experts *computed* in host RAM by llama.cpp (`-ot exps=CPU`, whence the link - carries activations only) and 25–26.5 on the iGPU. Streaming over PCIe loses; - computing in RAM does not care. FreeToken — expert LRU in host RAM, exactly - this streaming shape — also loads only modelopt block-FP8 safetensors for - `qwen3_5_moe`, so a Q4_K_M GGUF is unloadable there before bandwidth is even - the question. See [[x4-pair-experiment]]. +- **RCCL collectives need PCIe atomics on every participating GPU, and the failure never names + atomics.** The kernels carry a hidden hostcall-buffer parameter; ROCr refuses to dispatch them on + a device without atomics and the error surfaces as `hipErrorIllegalState` from `hipLaunchKernel`, + not as anything that names atomics. The value is consumed at driver PROBE time, so writing the + AtomicOp Requester bit with `setpci` at runtime changes nothing. Check the CHAIN: `lspci -vv -s + ` and every upstream bridge must show `AtomicOpsCap: Routing+` (a `Routing-` bridge is a + hardware wall), and the endpoint's Device Control 2 (`setpci -s CAP_EXP+28.w`) must have bit + 6 set. AMD desktop platforms put one card on CPU root ports (atomics work) and the rest behind the + chipset (usually not) — that asymmetry is what blocked TP=2 here. A one-card collective proves + nothing: NCCL short-circuits at `world_size=1`. [[PCIe Link and Atomics]] +- **That wall was physical, and a board swap removed it on 2026-09-20.** dankserver took the path of + a board whose CPU lanes bifurcate to x8/x8 Gen5, both cards on root ports, every bridge `Routing+` + and BOTH endpoints `ReqEn+` — the exact precondition the old test plan named. Bandwidth was never + the constraint (a 27B TP=2 allreduce at batch 4 is ≈10 MB per token-step, under 0.4 ms at Gen5 x8). + The collective choice matters: TheRock's RCCL builds lack gfx1201 device code + (`invalid device function`), which is why vLLM runs its R4D one-shot allreduce here at all. + Evidence matrix and the measured three-arm verdict: [[Dual-R9700-Cooperation]]. +- **When a GPU collective dies, read the runtime, not the framework.** The message that identified + the cause came from `AMD_LOG_LEVEL=3` (`Pcie atomics not enabled, hostcall not supported` → + `AQL dispatch failed!`), never from vLLM or RCCL — the env var costs one line and prints the + failing HIP call with its return code. [[PCIe Link and Atomics]] +- **Layer split is the no-collective path, on one vendor's cards and across two — and + `--override-tensor` is what cannot ride with it.** Layers go in order and only the boundary + activation crosses PCIe once per forward pass, so it works wherever a collective cannot be + dispatched (and on an RCCL-less fork it beats tensor mode, which falls back to a butterfly + allreduce); it is also the cross-vendor answer, where a CUDA + Vulkan split held a real + two-device graph. But `--override-tensor =` sets `has_tensor_overrides()`, a + REQUIRED term of the scheduler's `pipeline_parallel` flag (llama-context.cpp), and that flag sets + `n_copies` in `ggml_backend_sched_new`: with `n_copies == 1` a pre-allocated weight on a + non-primary device has no backend and the graph aborts (ggml-backend.cpp, reported with the leaf's + own op name `(NONE)`). `=CPU` is the only override target that works, because the CPU backend is + last in the list, and pure Vulkan changes nothing — the gate is the override, not the driver mix. + Pin the spec-decode drafter to one device: an unpinned draft under tensor mode aborts the KV build, + and on a split its home is the card holding the target's last layers. + [[Llama Cpp Multi-Device Splits]] [[x4-pair-experiment]] +- **A host↔3090 link at x4 measures 6.72 GB/s, and that is a design constraint, not a detail.** + Pinned H2D 6.72 / D2H 6.60 GB/s (≈85 % of the Gen4 x4 ceiling), async no better than sync and + pageable within 1 % of pinned — the copies are LINK-limited, not staging-limited, so the usual + "pageable costs 2×" rule does not apply. The card and the port beneath it are both wider than x4, + so the x4 lives in the adapter or cable, and the link idles at 2.5 GT/s until load. Consequence + for any engine that STREAMS weights across it: a 3 MiB slab costs 474 µs, so a 111 GiB A3B model + needing ~1.70 GiB of active experts per token lands at 3.7 tok/s, against 20.46 tok/s for the same + model with the experts COMPUTED in host RAM (`-ot exps=CPU`, where the link carries activations + only). Streaming over PCIe loses; computing in RAM does not care. [[x4-pair-experiment]] **ROCm / GPU** -- **Layer split needs no collective** — layers go in order and only the boundary - activation crosses PCIe, so it is the fallback wherever a collective cannot be - dispatched (and on an RCCL-less fork it beats tensor mode, which falls back to - a butterfly allreduce). Pin the spec-decode drafter to one device: an unpinned - draft under tensor mode aborts the KV build, and on a split its home is the - card holding the target's last layers. Deployed history: - [[frank-medium-llamacpp-dflash2]]. -- **Bundling a ROCm runtime next to a llama.cpp binary: `cp -r SRC DST/` is a - trap.** With a missing `DST`, cp creates `DST` as a copy of `SRC` and drops - one directory level. rocBLAS and hipBLASLt look for their device code at a - fixed depth (`../lib/rocblas/library`, `../lib/hipblaslt/library/`), - and the flattened copy fails at *request* time, not load time: the engine - reads the weights, answers `/health` and dies on the first completion with - `Cannot read .../TensileLibrary.dat: No such file or directory for GPU arch - : gfx1201`. Declare the destination, then copy the contents. -- **mmap must stay on** for the big-PLE models: `--no-mmap` forces the - ~50 GiB PLE table into anonymous memory → silent cgroup kill, dmesg is the - only evidence. mmap pages ARE page cache. Never pin the PLE tensor with - `-ot` (23 → 13.4 t/s). -- **HIP reads bare env vars** `TEMP`, `HOST`, `GPU_MAX_HEAP_SIZE`. Serve - scripts export env files with `set -a`, so a tunable named `TEMP` leaks - into HIP (`TEMP=1.0` sent the JIT cache to path `1.0` and segfaulted every - load, fixed 2026-08-13 by renaming to `TEMPERATURE`). Never name a tunable - these; check with +- **Bundling a ROCm runtime next to a llama.cpp binary: `cp -r SRC DST/` is a trap.** With a missing + `DST`, cp creates `DST` as a copy of `SRC` and drops one directory level. rocBLAS and hipBLASLt + look for their device code at a FIXED depth (`../lib/rocblas/library`, + `../lib/hipblaslt/library/`), so the flattened copy fails at REQUEST time, not load time: + the engine reads the weights, answers `/health` and dies on the first completion with + `Cannot read .../TensileLibrary.dat: No such file or directory for GPU arch : gfx1201`. Declare + the destination, then copy the contents. +- **mmap must stay on** for the big-PLE models: `--no-mmap` forces the ~50 GiB PLE table into + anonymous memory → silent cgroup kill, with dmesg as the only evidence. mmap pages ARE page cache, + and pinning the PLE tensor with `-ot` cost 23 → 13.4 t/s. [[Llama Cpp Serving Sizing]] +- **HIP reads bare env vars** `TEMP`, `HOST`, `GPU_MAX_HEAP_SIZE`. Serve scripts export env files + with `set -a`, so a tunable named `TEMP` leaks into HIP (`TEMP=1.0` sent the JIT cache to path + `1.0` and segfaulted every load, fixed 2026-08-13 by renaming to `TEMPERATURE`). Never name a + tunable these; check with `strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE"`. -- **`-DGGML_CUDA_FA_ALL_QUANTS=ON` is required** for q8_0 KV + flash - attention on CUDA builds; without it prefill collapses ~12x (95 vs 1300+ - t/s, measured 2026-08-17). -- **Flash-Next MTP heads do not load on any current loader** — they need full - per-layer `blk..hc_attn_norm.weight` and ship only `blk.48` (verified - 3x). Don't re-attempt without new loader support. -- **QSA flash-attn assert:** `GGML_ASSERT(a-type == GGML_TYPE_F32)` on - qwen4exp needs `scripts/patches/qwen4exp-qsa-f32-mask.patch` — but only on - the agentionai FAST layer set; the kingjones STRIX set works unpatched. -- **GPU budget on Dankclaw is the GTT aperture (~114 GiB), not the - rocm-smi "dedicated" figure.** The Radeon 8060S (gfx1151) reports - 116,732 MiB (`llama-server --list-devices`; - `gtt_total=122402963456` in `/sys/class/drm/card1/device/`). The older - "64 GiB, SVM-backed" lesson was already stale — the CIRU Orca predecessor - held 92.6 GiB of weights on the iGPU. `free`/`dmidecode` show the - CPU-side carve-out — intentional, do not change. +- **`GGML_CUDA_FA_ALL_QUANTS` is dead; do not build for it.** Verified on this host 2026-10-19: + master's `ggml/cmake/common.cmake` warns it is deprecated in favour of `GGML_CUDA_FA_QUANTS`, the + successor's DEFAULT pair list (`q4_0-q4_0;q8_0-q8_0;f16-f16;bf16-bf16`) already compiles the + quantized-KV kernels with the old flag OFF, and neither llama tree on this host references the old + macro anywhere under `ggml/src/ggml-cuda/`. The old "~12× prefill collapse" belongs to the older + source line. Prove a build's KV ability from its `CMakeCache.txt`. [[frank-zealot]] +- **Flash-Next MTP heads do not load on any current loader** — they need full per-layer + `blk..hc_attn_norm.weight` and ship only `blk.48` (verified 3×). Do not re-attempt without new + loader support. +- **QSA flash-attn assert:** `GGML_ASSERT(a-type == GGML_TYPE_F32)` on qwen4exp needs + `scripts/patches/qwen4exp-qsa-f32-mask.patch` — but only on the agentionai FAST layer set; the + kingjones STRIX set works unpatched. [[Llama Cpp Serving Sizing]] +- **The GPU budget on Dankclaw is the GTT aperture, not the rocm-smi "dedicated" figure.** The + Radeon 8060S (gfx1151) reports 116,732 MiB (`llama-server --list-devices`; `gtt_total` in + `/sys/class/drm/card1/device/`), and the older "64 GiB, SVM-backed" lesson was already stale. + `free`/`dmidecode` show the CPU-side carve-out — intentional, do not change. [[Dankclaw System Specs]] **Load / crash diagnosis** -- Model load is slow (minutes cold); health returns 503 while loading; do - not kill the process. `deploy.sh` waits up to 20 minutes (`LLM_WAIT`). On - halogen, the FIRST request after a (re)start can go silent ~45–60 s - (health `engine_unresponsive`, 503s) and then self-recovers — do not - restart over it. -- Crash-loop procedure: (1) `journalctl -u -n 100` + kernel line via - `journalctl -k` (segfault at address 8 in `libamdhip64` = NULL unref in - HIP); (2) run serve script, then raw binary — different outcome isolates - the env file; (3) `gdb -batch -ex run -ex "bt 25" --args bash - scripts/serve-.sh` — llama-server buffers stdout and loses the log - tail on SIGSEGV; trust gdb, not the log; (4) `amdgpu: SVM mapping failed` - spam is a symptom, not a root cause. -- Journal signatures: client abort logs `W srv stop: cancel task, id_task = - N`; natural stop logs only `slot release` with `truncated = 0`. The server - never preempts a running generation; excess tasks queue when slots are - busy. +- Model load is slow (minutes cold) and health returns 503 while it runs: do NOT kill the process, + `deploy.sh` waits up to 20 minutes (`LLM_WAIT`). On halogen the FIRST request after a (re)start + can go silent for ~45–60 s (`engine_unresponsive`, 503s) and then self-recovers — do not restart + over it. [[frank-dragoon]] +- Crash-loop procedure: (1) `journalctl -u -n 100` AND the kernel line via `journalctl -k` (a + segfault at address 8 in `libamdhip64` is a NULL unref in HIP); (2) run the serve script, then the + raw binary — a different outcome isolates the env file; (3) `gdb -batch -ex run -ex "bt 25" + --args bash scripts/serve-.sh`, because llama-server buffers stdout and loses the log tail on + SIGSEGV: trust gdb, not the log; (4) `amdgpu: SVM mapping failed` spam is a symptom, not a root + cause. [[LLM Performance Diagnosis]] +- Journal signatures: a client abort logs `W srv stop: cancel task, id_task = N` while a natural stop + logs only `slot release` with `truncated = 0`. The server never preempts a running generation; + excess tasks queue when slots are busy. **Host memory / swap** -- Swap resize ordering: create and `swapon` the replacement BEFORE `swapoff` - of the old one, or live pages must land in RAM that `MemAvailable` - overstates. Dankclaw swap (64 GiB at `/swap.img`, fstab-driven) protects - the co-tenants, not frank-dragoon — pinned weights don't swap, and paging - the KV pool is worse than the OOM it prevents. -- Host swap state is invisible to `deploy.sh` (no repo file) — check - `swapon --show` before reasoning about headroom. -- **A sysfs raise is a countdown to the next reboot.** `ttm.pages_limit` defaults - to half of RAM, so the manual raise to 200 GiB (2026-10-05) vanished with the - 2026-10-12 23:17 POWER FAILURE on dankserver: the cap fell to 125,988 MiB, - frank-archon's pinned-ledger preflight (floor 150,000 MiB) refused the start, - and `Restart=always` re-ran that refusal **6,919 times in ~20 h** — a unit in - `activating (auto-restart)` that exits 1 in 50 ms with no container and no - engine log is a PREFLIGHT refusal, not an engine fault. Repo law now: - `scripts/ttm-preflight.sh` (sourced by `serve-frank-archon.sh`) raises the knob - to `FRANK_ARCHON_TTM_TARGET_MIB` itself, re-reads, and refuses by name only - after a write the kernel rejected. Never prescribe a manual sysfs write as the - recovery procedure for a service that can boot before a human reaches it. -- **Silent RAM corruption outranks every software theory.** frank-archon's - ~40k-token output corruption (2026-10-04/05, repetition/garbage in long - agent conversations) was marginal DDR5 at 4800 MT/s on four DIMMs, fixed - by 4600. Consumer non-ECC memory raises **no MCE** for a silent flip, and - a short in-band pattern check (60 s, 8 GiB of 246 GiB) is not a stability - proof — combined, those two instruments said "ruled out" while the RAM - was the cause. "4800 is the JEDEC floor on 5600-rated DIMMs" is a rating, - not a margin guarantee. When the fault is data-shaped (NaN states, - garbage tokens) and a software revert doesn't move it, price in the RAM - before the runtime — and remember a revert tested under the same marginal - hardware is itself confounded. See the runbook's 2026-10-05 section. +- **Swap resize ordering: create and `swapon` the replacement BEFORE `swapoff` of the old one**, or + live pages must land in RAM that `MemAvailable` overstates. Dankclaw's 64 GiB `/swap.img` + (fstab-driven) protects the co-tenants, not an endpoint with pinned weights — paging the KV pool + is worse than the OOM it prevents. Host swap state is invisible to `deploy.sh` (no repo file): + `swapon --show` before reasoning about headroom. [[Swap Configuration]] +- **A sysfs raise is a countdown to the next reboot.** `ttm.pages_limit` defaults to half of RAM, so + a manual raise vanished with a power failure, frank-archon's pinned-ledger preflight refused the + start, and `Restart=always` re-ran that refusal 6,919 times in ~20 h — **a unit in + `activating (auto-restart)` that exits 1 in 50 ms with no container and no engine log is a + PREFLIGHT refusal, not an engine fault.** Repo law now: `scripts/ttm-preflight.sh` raises the knob + to `FRANK_ARCHON_TTM_TARGET_MIB` itself, re-reads, and refuses by name only after a write the + kernel rejected. Never prescribe a manual sysfs write as the recovery procedure for a service that + can boot before a human reaches it. [[frank-archon]] +- **Silent RAM corruption outranks every software theory.** frank-archon's long-conversation + repetition/garbage was marginal DDR5 at 4800 MT/s on four DIMMs, fixed by dropping to 4600. + Consumer non-ECC memory raises NO MCE for a silent flip, and a short in-band pattern check + (60 s over 8 GiB of 246 GiB) is not a stability proof — combined, those two instruments said + "ruled out" while the RAM was the cause, and "4800 is the JEDEC floor" is a rating, not a margin + guarantee. When the fault is data-shaped (NaN states, garbage tokens) and a software revert does + not move it, price in the RAM before the runtime — and remember a revert tested under the same + marginal hardware is itself confounded. [[log]] **Runtime-specific guards** -- **Rootless podman cannot RAISE rlimits, and `--ulimit nofile=1048576` - dies at container create, not at runtime.** runc reports - `error setting rlimit type 7: operation not permitted` (type 7 = - RLIMIT_NOFILE) when the requested hard limit exceeds the host process's - hard limit — on Dankclaw that ceiling is 524288. Inherited-equal limits - are fine, so `memlock=-1:-1` works iff the unit grants - `LimitMEMLOCK=infinity` (systemd is privileged; the value flows through - podman to runc without any raise). Upstream halogen's own run lines pass - NO `--ulimit` at all — copy vendor run lines before adding flags - (measured 2026-10-05 on the frank-dragoon cutover). -- **Kairic Edge** (if ever re-deployed): the pinned CK commit needs - `third_party/composable_kernel/include/ck/config.h` generated manually - (untracked; `git clean` removes it), and its greedy-argmax fastpath rejects - non-greedy requests and reasoning budgets. Re-test chat before production. -- **FastMTP sidecar**: build must be commit `4df29be4` + the HauhauCS patch; - stock binaries fail the draft load with a tensor-shape mismatch. -- **A GGUF's custom tensor type picks the runtime — read the histogram before - promising a file to a loader.** `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is - ftype 115 with 311 tensors of type 103 (`GGML_TYPE_Q8_0_ROCMFPX`); stock - llama.cpp and gufo refuse it at load, so ROCmFPX was not a preference but - the only door. Counting tensor types out of the GGUF KV section takes one - python read and settles a runtime argument in minutes. The build identity - matters as much as the fork: the 2026-08-25 ROCmFPX builds SIGSEGV in - `ggml_compute_forward_set_rows` on this same file, so - `scripts/build-rocmfpx.sh` pins commit `36e9acd` by SHA. -- **Speculative-decode flags move as a SET.** Handing the ROCmFPX fork - `--model-draft` with no `--spec-type` (the naive way to build a "no MTP" - bench arm) segfaulted at load, core dumped, no diagnostic (2026-10-02). A - baseline arm must strip `--model-draft` AND every `--spec-draft-*` AND - `--spec-type`, or it measures a crash instead of the absence of drafting. -- **An orphaned scratch server taxes the whole host.** A bench child left - alive by a client-side exception kept ~26 GiB of weights resident on a - 121 GiB box, pushed `available` to 3 GiB, and every later rate was suspect. - Record the child pid and kill it in a `finally`, then prove `free` came back - before believing a number. Kill by pid — never `pkill -f llama-server`, - which also hits frank-zealot. +- **Rootless podman cannot RAISE rlimits, so `--ulimit nofile=1048576` dies at container CREATE**, + not at runtime: runc reports `error setting rlimit type 7: operation not permitted` (type 7 = + RLIMIT_NOFILE) when the requested hard limit exceeds the host process's ceiling (524288 on + Dankclaw). Inherited-equal limits are fine, so `memlock=-1:-1` works iff the unit grants + `LimitMEMLOCK=infinity` — systemd is privileged and the value flows through podman to runc without + any raise. Upstream halogen's own run lines pass NO `--ulimit` at all: copy vendor run lines + before adding flags. [[frank-dragoon]] +- **A GGUF's custom tensor type picks the runtime — read the histogram before promising a file to a + loader.** `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is ftype 115 over 311 tensors of type 103 + (`GGML_TYPE_Q8_0_ROCMFPX`), so stock llama.cpp and gufo refuse it at load and the fork was not a + preference but the only door. Counting tensor types out of the GGUF KV section takes one python + read and settles a runtime argument in minutes. Build identity matters as much as the fork: the + 2026-08-25 ROCmFPX builds SIGSEGV in `ggml_compute_forward_set_rows` on this same file, so + `scripts/build-rocmfpx.sh` pins commit `36e9acd` by SHA. [[GGUF Compatibility]] +- **Speculative-decode flags move as a SET.** Handing the fork `--model-draft` with no `--spec-type` + (the naive way to build a "no MTP" bench arm) segfaulted at load, core dumped, no diagnostic. A + baseline arm must strip `--model-draft` AND every `--spec-draft-*` AND `--spec-type`, or it + measures a crash instead of the absence of drafting. [[frank-zealot]] +- **An orphaned scratch server taxes the whole host.** A bench child left alive by a client-side + exception kept ~26 GiB of weights resident on a 121 GiB box and pushed `available` to 3 GiB, so + every later rate was suspect. Record the child pid, kill it in a `finally`, then prove `free` came + back before believing a number — and kill by pid, never `pkill -f llama-server`. + [[LLM Bench Windows and Measurement]] +- Retired-runtime traps that can come back: **Kairic Edge** needs + `third_party/composable_kernel/include/ck/config.h` generated manually (untracked, so `git clean` + removes it) and its greedy-argmax fastpath rejects non-greedy requests and reasoning budgets, so + re-test chat before production; the **FastMTP sidecar** must be commit `4df29be4` + the HauhauCS + patch, since stock binaries fail the draft load with a tensor-shape mismatch. **llama.cpp serving** -- **MTP is a property of the FILE, not of the flags.** The `-mtp` GGUF build - carries the head as `blk..nextn.*` (`qwen35.nextn_predict_layers = 1`); - `--spec-type draft-mtp` then sets `load_mtp`, and the draft context runs on - the MAIN model's weights (`shares_model`), so there is no second copy of a - 12 GiB model in VRAM and no `-md` argument. The plain build has no MTP - tensors at all, and pointing `--model-draft` at the file would load it twice. -- **What can be chained, what cannot, and what a chain costs on a hybrid.** - `--spec-type a,b` builds every enabled implementation and chains drafting - through them, in a fixed internal order (ngram types first, then the - drafters — CLI order is irrelevant). The gate is `ctx_dft`: **draftless** - types (the ngram family) are added unconditionally, while `draft-mtp`, - `draft-dflash`, `draft-dspark` and `draft-eagle3` all require a draft - CONTEXT. So a chain may hold one context-backed drafter plus any number of - ngram types — `draft-dflash,ngram-mod` is legal and measured (2.6x the - drafter alone on prompt-echo traffic, byte-identical to it on prose at the - fleet's 48/64 window) — but `draft-mtp,draft-dflash` dies at load: *"context - type MTP requested but model doesn't contain MTP layers"*, because MTP - builds its context against whatever model the draft context holds. Two - context-backed types cannot share one process, and MTP's head only loads - when MTP **is** the draft model. **A LEGAL chain can still lose on a - recurrent model:** `draft-mtp,ngram-mod` on frank-dragoon's qwen35 hybrid - measured −15 % prose / −7 % block-copy (2026-10-03) — adding the ngram layer - logs `recurrent state rollback is not compatible with 'ngram-mod' - - disabling rollback support`, and MTP verification depends on that rollback, - so acceptance drops on ALL traffic even where the matcher fires. Measure - chains per architecture, never by analogy. -- **A drafter's KV scales with the TARGET's context, and its window policy - decides the price.** The draft context is built at `llama_n_ctx(ctx_tgt)` with - no flag to cap it, so a drafter declaring NO sliding window caches the whole - window on `--spec-draft-device` and can OOM the first draft graph capture - AFTER `/health` has answered. Read `sliding_window` out of the drafter's GGUF - header before sizing, and prefer a skewed `-ts` over a context cut. Worked - example: [[frank-medium-llamacpp-dflash2]]. -- **A draft context bills its own f16 KV over the WHOLE POOL, and on a - single-card endpoint that squeeze can cost more than the drafter wins.** With - `--parallel N` the drafter pays the pool, not the slot, plus its own compute - buffers. So the MTP question is never "does the file carry the head" but "what - do draft KV + buffers cost against what the batch knobs leave for compute" — - a per-shape answer. Measure the arm, never the flag; the booted arm table is - on [[frank-zealot]]. -- **A quant's scale overhead belongs in its byte rate; doubling a measured - figure to "get the f16 number" invents 6.25 %.** q8_0 is 1.0625 B/element - (32 codes + one 2-byte scale per 32-block), f16 is exactly 2.0 ⇒ the ratio is - **1.882×**, not 2×. Derive byte rates from element count × block layout, then - check them against the boot ledger — this repo carried an invented 2× number - for weeks. Current KV math: [[frank-zealot]]. -- **`--ubatch-size` has an OPTIMUM, not a plateau, and the curve is - BACKEND-SPECIFIC — never carry it across hosts.** Its compute buffers land - about TWICE as heavy on the card that holds the output layers and the drafter, - so a split balanced at one batch is unbalanced at another: re-measure the - balance after any batch change. The logical `--batch-size` is nearly free. - A large-batch allocation can fail on BUFFER CONTIGUITY while totals still fit - — no totals model can see that. Ladders: [[frank-medium-llamacpp-dflash2]] - (R9700 split) and [[frank-zealot]] (CUDA 3090). -- **On the CUDA 3090 that same knob has no interior optimum up to 2048.** A - prompt shorter than the ubatch cannot see the knob at all — that is the - control that proves the knob, not the traffic, moved the number. When a bench - IS ordered, A/B it by restarting the SERVICE through a transient - `FRANK_ZEALOT_ENV_FILE` drop-in, never a scratch port: one card holds one - server. Numbers: [[frank-zealot]]. -- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same - size** (a DeltaNet boundary and a full-attention boundary can differ by ~1 GiB - each with its own KV), so balance by measurement. An affine fit reproduces the - deployed point and overstates the gaps in between. - [[frank-medium-llamacpp-dflash2]]. -- **ngram-mod prints no startup banner.** The load log shows the drafter's - `adding speculative implementation 'draft-dflash'` and nothing at all for the - matcher, so a chain can look unengaged while it is working. Identify it - behaviourally (`draft_n` > 0 only where text repeats) or by the per-task - `draft acceptance = ... mean len = ...` line. It costs no VRAM (a 64 MiB - host-RAM hash table) and its window decides which traffic it can see: too - short and it proposes on ordinary prose fragments, which costs verification - without acceptance. -- **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the - per-slot number is the contract — not `ctx-size`.** `n_ctx_slot = - CTX_SIZE / PARALLEL`, and llama.cpp refuses a prompt longer than the SLOT with - a 400 even when the pool has room, so `models.yml` `contextWindow` must follow - every reslice. Both slot directions cost or buy window, never pool; shorter - slots decode slightly faster (less attention to scan, one fewer recurrent - state). Measured reslices: [[frank-zealot]]. -- **Price a VRAM budget from two boots, never from a formula — and an mmproj - costs ~1.5× its file size** (the encoder's activation and compute buffers ride - with it). Subtracting two booted rows isolates it exactly; extrapolating - weights+KV+"a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7. - Then solve `KV ≤ card − non-KV − headroom` and round to a multiple of - `PARALLEL × 4096`. The two-boot table: [[frank-zealot]]. -- **`images` and `image_url` are NOT interchangeable: only the content part - reaches the model.** `--mmproj` loads, `usage.prompt_tokens` rises, and the - model still invents an answer if the client used the legacy top-level - `images` field on `/v1/chat/completions` — frank-zealot answered "Brown." for - a solid-red PNG that way and "Red." through the OpenAI `image_url` part. The - failure is silent because a hallucinated color is a plausible answer. Probe - vision with a solid color and expect an exact word. -- **`--jinja` is what `tools` needs.** On 2026-09-16 a subagent died with - `500 tools param requires --jinja flag`, from the ik_llama.cpp runtime that - then served frank-dragoon: every tool-using agent on that endpoint was broken - and the failure read like a model fault. Mainline llama.cpp 0.4.1 defaults - `--jinja` on and the serve script pins it; halogen also accepts `tools` and - returns `tool_calls`. Check tool support before routing a role at an endpoint. -- **Quantized KV on CUDA needs `GGML_CUDA_FA_ALL_QUANTS=ON`** — the same - option and the same ~12x prefill collapse described in the ROCm note above. - f16 KV needs nothing; frank-zealot ran f16 for that reason until - 2026-09-17, when `build-cuda` was rebuilt with the option ON so the - endpoint could take q8_0 KV. -- **llama-bench separates a LIST with `/` and SWEEP values with `,`.** Both - `-dev` and `-ts` follow this: `-dev CUDA0,Vulkan1` is two *single-device* - configurations, never one two-device run, and `-ts 0.85,0.15` is two sweep - values each becoming `[0.85, 0, …]` / `[0.15, 0, …]`, i.e. device 0 takes - everything. Write `-dev A/B -ts x/y`. This artifact produced a wrong fleet - conclusion on 2026-09-16 — "CUDA + Vulkan cannot serve one model", with two - bogus OOMs that were really "all 111 GiB on the 24 GiB card" — and the bench - table's own `dev` column was printing the truth the whole time. Any - placement-dependent number needs an independent check: sample the other +- **MTP is a property of the FILE, not of the flags.** The `-mtp` export carries the head as + `blk..nextn.*` (`qwen35.nextn_predict_layers = 1`); `--spec-type draft-mtp` sets + `load_mtp` and the draft context runs on the MAIN model's weights (`shares_model`), so there + is no second copy of a 12 GiB model in VRAM and no `-md` argument. The plain build has no MTP + tensors at all, and pointing `--model-draft` at that file loads it twice. [[frank-zealot]] +- **What can be chained: ONE context-backed drafter plus any number of draftless ngram types.** + `--spec-type a,b` chains in a fixed internal order (CLI order irrelevant) and the gate is + `ctx_dft`: the ngram family is added unconditionally while `draft-mtp`, `draft-dflash`, + `draft-dspark` and `draft-eagle3` all need a draft CONTEXT. So `draft-dflash,ngram-mod` is + legal (measured 2.6x the drafter alone on prompt-echo traffic) but `draft-mtp,draft-dflash` + dies at load — *"context type MTP requested but model doesn't contain MTP layers"*. **A LEGAL + chain can still lose on a recurrent model:** `draft-mtp,ngram-mod` on frank-dragoon's qwen35 + hybrid measured −15 % prose / −7 % block-copy, because adding the ngram layer logs + `recurrent state rollback is not compatible with 'ngram-mod' - disabling rollback support` and + MTP verification depends on that rollback. Measure chains per architecture, never by analogy. + [[frank-dragoon]] +- **A drafter's KV scales with the TARGET's context, and its window policy decides the price.** + The draft context is built at `llama_n_ctx(ctx_tgt)` with no flag to cap it, so a drafter + declaring NO sliding window caches the whole window on `--spec-draft-device` and can OOM the + first draft graph capture AFTER `/health` has answered. Read `sliding_window` out of the + drafter's GGUF header before sizing, and prefer a skewed `-ts` over a context cut. + [[Llama Cpp Multi-Device Splits]] +- **A draft context bills its own f16 KV over the WHOLE POOL, not the slot**, plus its own + compute buffers, so on a single-card endpoint that squeeze can cost more than the drafter + wins. The MTP question is never "does the file carry the head" but what draft KV + buffers + cost against what the batch knobs leave for compute. Measure the arm, never the flag. + [[frank-zealot]] +- **A quant's scale overhead belongs in its byte rate; doubling a measured figure to "get the + f16 number" invents 6.25 %.** q8_0 is 1.0625 B/element (32 codes + one 2-byte scale per + 32-block), f16 is exactly 2.0 ⇒ the ratio is **1.882×**, not 2×. Derive byte rates from element + count × block layout, then check them against the boot ledger — this repo carried an invented + 2× number for weeks. Current KV math: [[frank-zealot]] +- **`--ubatch-size` has an OPTIMUM, not a plateau, and the curve is BACKEND-SPECIFIC — never + carry it across hosts.** Its compute buffers land about TWICE as heavy on the card that holds + the output layers and the drafter, so a split balanced at one batch is unbalanced at another: + re-measure the balance after any batch change. The logical `--batch-size` is nearly free; a + large-batch allocation can fail on BUFFER CONTIGUITY while totals still fit, which no totals + model can see. A prompt shorter than the ubatch cannot see the knob at all — that is the + control that proves the knob, not the traffic, moved the number. The 2× R9700 split peaks at + 512; the CUDA 3090 has no interior optimum up to 2048. When a bench IS ordered, A/B it by + restarting the SERVICE through a transient `FRANK__ENV_FILE` drop-in, never a scratch + port: one card holds one server. Ladders: [[frank-medium-llamacpp-dflash2]] and + [[frank-zealot]]. +- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same size** (a DeltaNet + boundary and a full-attention boundary can differ by ~1 GiB each with its own KV), so balance + by measurement. An affine fit reproduces the deployed point and overstates the gaps in between. + [[Llama Cpp Multi-Device Splits]] +- **`ngram-mod` prints no startup banner.** The load log names `draft-dflash` and nothing for the + matcher, so a chain can look unengaged while it works. Identify it behaviourally (`draft_n` > 0 + only where text repeats) or by the per-task `draft acceptance = ... mean len = ...` line. It + costs no VRAM (a 64 MiB host-RAM hash table) and its window decides which traffic it can see: + too short and it proposes on ordinary prose fragments, costing verification without acceptance. + [[Llama Cpp Multi-Device Splits]] +- **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the per-slot number is + the contract — not `ctx-size`.** `n_ctx_slot = CTX_SIZE / PARALLEL`, and llama.cpp refuses a + prompt longer than the SLOT with a 400 even when the pool has room, so `models.yml` + `contextWindow` MUST follow every reslice. Both slot directions cost or buy window, never + pool; shorter slots decode slightly faster. Measured reslices: [[frank-zealot]]. +- **Price a VRAM budget from two boots, never from a formula — and an mmproj costs ~1.5× its file + size** (the encoder's activation and compute buffers ride with it). Subtracting two booted rows + isolates it exactly; extrapolating weights+KV+"a bit for buffers" is what leaves 1.1 GiB free + when you meant 2.7. Then solve `KV ≤ card − non-KV − headroom` and round to a multiple of + `PARALLEL × 4096`. **The projector is NOT billed in llama.cpp's own fit check**: the engine + prints its worst case and fits WITHOUT it, so price a shape from settled `nvidia-smi` after a + real boot — the projector, and any drafter context, lands after `/health` would already have + answered (same class as the DSpark drafter OOM). Two-boot table: [[frank-zealot]]. +- **`images` and `image_url` are NOT interchangeable: only the content part reaches the model.** + `--mmproj` loads, `usage.prompt_tokens` rises, and the model still invents an answer if the + client used the legacy top-level `images` field on `/v1/chat/completions` — frank-zealot + answered "Brown." for a solid-red PNG that way and "Red." through the OpenAI `image_url` part. + The failure is silent because a hallucinated color is a plausible answer; probe with a solid + color and expect an exact word. [[frank-zealot]] +- **`--jinja` is what `tools` needs.** A subagent died with `500 tools param requires --jinja + flag` on the ik_llama.cpp runtime that served frank-dragoon: every tool-using agent on that + endpoint was broken and the failure read like a model fault. Mainline llama.cpp defaults + `--jinja` on and the serve scripts pin it; halogen also accepts `tools` and returns + `tool_calls`. Check tool support before routing a role at an endpoint. +- **A model's own template can REJECT the harness's thinking level.** Qwen3.8's template accepts + only `xhigh`, `medium` and `low` for `reasoning_effort` and raises on anything else — HTTP 500, + not a clamp — so the harness's levels must be folded onto that set in `models.yml` + (`compat.reasoningEffortMap`). A role whose level is unmapped fails on its FIRST request and + the failure names Jinja, not the model: every subagent on `frank-zealot` died with + `Unexpected reasoning effort high` until `high: xhigh` was added. Halogen accepts the wider + set, so the two entries legitimately differ. +- **Thinking tokens count against `max_tokens`, so a short-budget vision probe reads as + blindness.** A probe at `max_tokens: 64` returned `content: ""` with `finish_reason: "length"` + and half a thought in `reasoning_content` while the image HAD reached the model (322 prompt + tokens vs 42 for the text alone). Budget ≥512 for any thinking model you probe for vision. +- **The FIRST request after a boot is a warmup artifact**, and the 3× spread on one shape makes + graph-capture cost read as a model fault. Benchmark the second request or later. + [[frank-zealot]] +- **A fork build hides its own dtype laws.** Vendor builds carry their own compile options, so + prove a build's KV behaviour from the KV buffer line in the boot ledger and its + `CMakeCache.txt` — never from the build flags you remember passing. [[frank-zealot]] +- **A llama.cpp endpoint is invisible to Prometheus unless `--metrics` is passed**: without it + the engine answers **501 Not Implemented** on `/metrics`, so the scrape fails while `/health` + answers 200 and `LlmEndpointDown` fires beside a healthy service — that is how `frank-zealot` + sat red across four payload generations. halogen serves `/metrics` natively, so the asymmetry + reads as "only the 3090 is broken". Diagnose with `lastError` from `/api/v1/targets`, not the + alert list, and note Alertmanager has no persistence: a reboot re-stamps `startsAt` on an alert + that has fired for weeks. [[Observability]] +- **A leftover bench drop-in silently hijacks the next deploy, and no health gate sees it.** The + sanctioned bench-window method (a transient `Environment=FRANK__ENV_FILE=/tmp/.env` + drop-in sourcing the repo env) outlives the bench unless retired: a leftover + `frank-zealot.service.d/ptq1.conf` made the cutover deploy boot a **different model at a + different shape** while `/health` answered 200, `/v1/models` echoed the alias (a flag, not a + fact) and the Playwright gate passed 33/33. After deploying any recently-benched LLM service: + `systemctl show -p DropInPaths` (must be empty), then + `tr '\0' ' ' -/` + `daemon-reload`. [[frank-zealot]] +- **A bench verdict that never reaches the repo is indistinguishable from a rogue arm.** That + same arm served production for 24 minutes with real numbers (~48 t/s decode, 801 t/s prefill at + 141k) that lived only in `/tmp` env files and journald, so the next agent read the surviving + drop-in as an unfinished experiment and reported a hijack. Write the measurement into the + endpoint page and the arm script's header in the session that takes it — `/tmp` and the journal + are not records. +- **`llama-bench` separates a LIST with `/` and SWEEP values with `,`** — for both `-dev` and + `-ts`. `-dev CUDA0,Vulkan1` is two *single-device* configurations, never one two-device run, + and `-ts 0.85,0.15` is two sweep values each putting everything on device 0. Write + `-dev A/B -ts x/y`. This artifact produced a wrong fleet conclusion ("CUDA + Vulkan cannot serve + one model") from two bogus OOMs, and the bench table's own `dev` column was printing the truth + the whole time. Any placement-dependent number needs an independent check: sample the other device's memory (`nvidia-smi`) while the run is loaded. -- **A model's own template can REJECT the harness's thinking level.** Qwen3.8's - template accepts only `xhigh`, `medium` and `low` for `reasoning_effort` and - raises on anything else — HTTP 500, not a clamp — so the harness's levels - must be folded onto that set in `models.yml` (`compat.reasoningEffortMap`). - A role whose level is unmapped fails on its FIRST request, and the failure - names Jinja, not the model: every subagent on `frank-zealot` died with - `Unexpected reasoning effort high` until `high: xhigh` was added on - 2026-09-16. Halogen accepts the wider set, so the two entries legitimately - differ. -- **The vision projector is NOT billed in llama.cpp's own fit check.** The - engine prints `[mtmd] estimated worst-case memory usage of mmproj is - 873.10 MiB` and then fits WITHOUT it: frank-zealot's Bonsai-2 boot projected - 21,439 MiB and settled at 22,664 (2026-10-17). Price a 3090-class shape from - settled `nvidia-smi` after a real boot, never from - `model + context + compute` — the projector (and any drafter context) lands - after `/health` would already have answered. Same failure class as the - DSpark drafter OOM. -- **The FIRST request after a boot is a warmup artifact.** Measured on - Bonsai-2/3090: 2.6 t/s prefill and 37.7 t/s decode on request #1, then - 1,258 t/s prefill and 54.7 t/s (per stream, two concurrent) on the next — a - 3× spread on the same shape. Benchmark the second request or later, or the - graph-capture cost reads as a model fault. -- **Thinking tokens count against `max_tokens`, so a short-budget vision probe - reads as blindness.** The Bonsai-2/`image_url` probe at `max_tokens: 64` - returned `content: ""` with `finish_reason: "length"` and half a thought in - `reasoning_content` — the image HAD reached the model (322 prompt tokens vs - 42 for the text alone). At 512 tokens the same image answers "Red". Budget - ≥512 for any thinking model you probe for vision. -- **Quantized KV is not the only dtype law a fork build can hide.** Vendor - builds carry their own compile options: frank-zealot's PrismML prebuilt - served `--cache-type-v q8_0` under flash attention at the exact 34,816 - B/token (the `GGML_CUDA_FA_ALL_QUANTS` door), so a build decision must be - proven by the KV buffer line in the boot ledger, not by the build flags you - remember passing. -- **A llama.cpp endpoint is invisible to Prometheus unless `--metrics` is passed.** - Without it the engine answers **501 Not Implemented** on `/metrics`, so the scrape - fails while `/health` answers 200 and `LlmEndpointDown` fires beside a healthy - service — that is how `frank-zealot` sat red across four payload generations - (the oldest retained sample of `up{job="frank-zealot"}`, 2026-10-11, is already - 0). halogen serves `/metrics` natively, so the asymmetry reads as "only the 3090 - is broken". Diagnose with `lastError` from `/api/v1/targets`, not the alert list, - and note Alertmanager has no persistence: a reboot re-stamps `startsAt` on an - alert that has been firing for weeks. -- **A leftover bench drop-in silently hijacks the next deploy, and no health - gate sees it.** The sanctioned bench-window method — a transient - `Environment=FRANK__ENV_FILE=/tmp/.env` drop-in, the arm env - `source`ing the repo env and then overriding it — outlives the bench unless - it is retired. On 2026-10-18 a leftover `frank-zealot.service.d/ptq1.conf` — - the arm Frank had ordered at 02:36 and measured from 03:14 — made the cutover - deploy boot a **different model at a different shape** (3 slots of 133,888, - not 3 × 262,144) while `/health` answered 200, `/v1/models` echoed the alias - (a flag, not a fact) and the Playwright gate passed 33/33. After deploying any - recently-benched LLM service: `systemctl show -p DropInPaths` (must be - empty), then `tr '\0' ' ' -/` + - `daemon-reload`. -- **A bench verdict that never reaches the repo is indistinguishable from a - rogue arm.** Same window: PTQ1_0 served production for 24 minutes with real - numbers (~48 t/s decode, 801 t/s prefill at 141k), but the figures lived only - in `/tmp` env files and journald, so the next agent on the host read the - surviving drop-in as an unfinished experiment and reported a hijack. Write the - measurement into the runbook and the arm script's header in the session that - takes it — `/tmp` and the journal are not records. **Ops** -- Never `pkill -f` a pattern that appears in your own command line — it kills - your ssh session. Use `pkill -x `. -- Both hosts allow passwordless sudo for the deploy user. Use plain `sudo` in - committed scripts (no `echo pw | sudo -S`); `FRANK_SUDO_PW` covers hosts - without it. -- ComfyUI on Dankclaw runs with **GPU access** (`/dev/kfd` + `/dev/dri` via - `AddDevice`, `GroupAdd=keep-groups`, plus the udev rule that makes those nodes - world-writable — user-unit spawns carry no supplementary groups). It must NOT - pass `--cpu`: the gfx1151 ROCm wheels exist and the unit's `--cpu` era ended - 2026-09-16, when a stale device-less container was found restarted 9,146 - times (`RuntimeError: No HIP GPUs are available`). -- SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its - ports off the gluetun service. -- Dankclaw boots **`iommu=pt`** since 2026-10-18. The boot line is tracked in - `dankclaw/host/grub` and moves ONLY through `scripts/set-dankclaw-bootline.sh` - (diff, install, `update-grub`, verify `grub.cfg`, `--reboot`) — never by - hand-editing `/etc/default/grub`. It replaced `amd_iommu=off` (2026-09-11), - adopted on the vendor's 13–16 % prefill figure and never measured here: the - local A/B on byte-identical traffic says identity mode costs **nothing** - (+2.0 %, 1,616 t/s cold prefill), so the XDNA2 NPU returned for free — - `amdxdna` binds `c6:00.1` (PASID width 10, identity group 26) and the old - `err=-19` only ever asked for an enabled IOMMU. **AMD PerfOpt** (iommu tree - 2026-09-24, Linux 7.4) still skips the page walk outright for the privileged - iGPU, but it arms only in the identity domain and 26.04 caps at `7.0.0-38`. -- Rootless Quadlet containers run with `--replace --rm`: `podman logs` covers - only the CURRENT instance; history across recreations lives in - `journalctl --user -u .service`. Read rootless storage WITHOUT sudo - (per-user storage — `sudo podman logs` finds nothing). -- **A dead rootless podman pause process breaks ALL container management on the - host, and `podman system migrate` panics in 5.7.0.** Symptoms: every `podman` - command fails with `invalid internal status … could not find any running - process`, every container start fails binding podman's bridge addresses - (`Error starting server failed to bind udp listener on 10.89.x.1:53: Cannot - assign requested address`), and every `Restart=always` quadlet loops. Container - storage is intact; what is gone is `/run/user//libpod/tmp/alive` (a UUID - token, not a pid) and the rootless netns holding the bridges — deleting the - stale token is not enough, and `podman system migrate` segfaults in - `teardownNetworkBackend`. Repair: **reboot** (`/run/user` clears and all 28 - quadlets return on their own — check `Linger=yes` and `WantedBy=default.target` - first). Seen 2026-09-16, hours into heavy GPU work on the same host — but the - trigger is NOT established (`journalctl -k` recorded no OOM kill; the kernel - had no other word on it). Treat the pause process as host state that can die - for its own reasons, and reach for the reboot when it does. -- **A container cannot reach a `Network=host` service at the host's LAN IP.** - netavark DNATs *published* ports only: from a bridge container - `192.168.0.143:` answers while `192.168.0.143:` and the - gateway `10.89.1.1` are both refused. Use `host.containers.internal` - (169.254.1.2). This is what 502'd `hermes.franksdank.cloud` until the Caddyfile - upstream was changed on 2026-09-16 (its vhost comment had claimed the LAN IP - was open from that bridge — it never was). -- **A `*.gguf` filter misses mradermacher's payloads: large quants ship as - `.partNofM` fragments.** `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF` - carries Q4_K_M (37.00 + 37.00 + 36.97 = 110.97 GiB) and Q4_K_S (35.00 + 35.00 - + 34.09 = 104.09 GiB) as parts, while the smaller rungs (Q2_K, Q3_K_*, IQ3_*, - IQ4_XS) are single files. Fetch the parts, verify each against the Hub's LFS - sha256, concatenate in numeric order — that is how the deployed 111 GiB file - was assembled. Listing the Hub tree and filtering `endswith(".gguf")` makes - Q4_K_S look like it does not exist. -- **A fetch script must `-f` and verify, or it reports success on an access - error.** `curl -o FILE URL` exits 0 on HTTP 401/403 and writes the error body - as the "artifact": a gated repo yielded three 153-byte shards with identical - sha256 values and a cheerful `DOWNLOAD_DONE` in 20 seconds for 104 GiB. Use - `-f`/`--fail-with-body`, compare each file's byte count to the Hub's declared - size, and check the LFS sha256 — `scripts/download-frank-medium-gguf.sh` - already does all three; ad-hoc fetches tend not to. Gated repos (e.g. - `orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF`, same 104.09 GiB Q4_K_S) - return 403 even with the host's HF token until that account accepts the gate. -- **A scout verdict is a lead, never a finding.** In the 2026-10-19 vault de-dup - campaign, four read-only scouts worked the same pages: one classified era pages as - stale *because* they document retired payloads (that is an era page's content), one - reported files that do not exist, one reported duplications that grep disproved - (`457.9 t/s` and the ping-RTT table are solo-held by `Ethernet Bandwidth Math`), - and one cited a line number past the end of the file it judged. Re-verify every - claimed duplicate and every claimed staleness against the file and today's fleet - state before editing; a scout yield buys no evidence. +- Never `pkill -f` a pattern that appears in your own command line — it kills your ssh session. + Use `pkill -x `. +- Both hosts allow passwordless sudo for the deploy user. Use plain `sudo` in committed scripts + (no `echo pw | sudo -S`); `FRANK_SUDO_PW` covers hosts without it. +- **ComfyUI on Dankclaw runs with GPU access** (`/dev/kfd` + `/dev/dri` via `AddDevice`, + `GroupAdd=keep-groups`, plus the udev rule that makes those nodes world-writable — user-unit + spawns carry no supplementary groups) and must NOT pass `--cpu`: the gfx1151 ROCm wheels exist + and a stale device-less container was found restarted 9,146 times. [[ComfyUI]] +- SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its ports off the gluetun + service. [[SearXNG]] [[Gluetun]] +- **Dankclaw's kernel command line is a repo file.** It boots **`iommu=pt`** since 2026-10-18, + tracked in `dankclaw/host/grub`, and moves ONLY through `scripts/set-dankclaw-bootline.sh` + (diff, install, `update-grub`, verify `grub.cfg`, `--reboot`) — never by hand-editing + `/etc/default/grub`. It replaced `amd_iommu=off`, and the local A/B on byte-identical traffic + says identity mode costs nothing, so the XDNA2 NPU returned for free. + [[dankclaw-iommu-translation-off]] +- **Rootless Quadlet containers run with `--replace --rm`**: `podman logs` covers only the CURRENT + instance; history across recreations lives in `journalctl --user -u .service`. Read + rootless storage WITHOUT sudo — it is per-user storage, so `sudo podman logs` finds nothing. + [[Quadlet Lifecycle]] +- **A dead rootless podman pause process breaks ALL container management on the host, and + `podman system migrate` panics in 5.7.0.** Every `podman` command fails with + `invalid internal status`, every start fails binding the bridges + (`failed to bind udp listener on 10.89.x.1:53`), every `Restart=always` quadlet loops — while + container storage stays intact. What is gone is `/run/user//libpod/tmp/alive` (a UUID + token) and the rootless netns; deleting the stale token is not enough. Repair: **reboot** + (`/run/user` clears and the quadlets return on their own — check `Linger=yes` and + `WantedBy=default.target` first). The 2026-09-16 occurrence had no OOM in `journalctl -k`: + treat the pause process as host state that can die for its own reasons. [[Quadlet Lifecycle]] +- **A container cannot reach a `Network=host` service at the host's LAN IP**: netavark DNATs + *published* ports only, so the host bind and the `10.89.1.1` gateway are both refused from a + bridge container while the published port answers. Use `host.containers.internal` + (169.254.1.2). This 502'd `hermes.franksdank.cloud` until its Caddyfile upstream changed. + [[Container Host Reachability]] +- **A `*.gguf` filter misses mradermacher's payloads: large quants ship as `.partNofM` + fragments** while the smaller rungs are single files, so filtering `endswith(".gguf")` makes a + whole quant class look absent. Fetch the parts, verify each against the Hub's LFS sha256, + concatenate in numeric order. [[Model Swap and Rollback]] +- **A fetch script must `-f` and verify, or it reports success on an access error.** + `curl -o FILE URL` exits 0 on HTTP 401/403 and writes the error body as the "artifact": a gated + repo yielded three 153-byte shards with identical sha256 and a cheerful `DOWNLOAD_DONE` in 20 + seconds for 104 GiB. Use `-f`/`--fail-with-body`, compare each file's byte count to the Hub's + declared size, and check the LFS sha256 (`scripts/download-frank-medium-gguf.sh` does all three; + ad-hoc fetches tend not to). Gated repos 403 even with the host's HF token until that account + accepts the gate. [[Model Swap and Rollback]] +- **A scout verdict is a lead, never a finding.** In the 2026-10-19 vault de-dup campaign six + read-only scouts worked the same pages: one classified era pages as stale *because* they + document retired payloads (that is an era page's content), one reported files that do not exist, + one reported duplications that grep disproved (`457.9 t/s` and the ping-RTT table are solo-held + by `Ethernet Bandwidth Math`), one cited a line number past the end of the file it judged, and + one drafted an Eras table with invented windows, payloads and rollback paths. Re-verify every + claimed duplicate, every claimed staleness and every quoted holder against the file and today's + fleet state before editing; a scout yield buys no evidence. ## Runbook Index diff --git a/scripts/check-okf.py b/scripts/check-okf.py new file mode 100755 index 0000000..f14dfa6 --- /dev/null +++ b/scripts/check-okf.py @@ -0,0 +1,175 @@ +#!/usr/bin/env python3 +"""OKF v0.1 conformance linter for the docs vault (serverAppConfigs-docs). + +The vault is an Open Knowledge Format bundle (omp-docs/references/okf-spec.md) +plus the maintenance discipline in okf-maintenance.md, plus this repo's own +index law (AGENTS.md -> *Documentation Policy*): an index is a locator, one row, +one clause, no numbers. Enforced here so a violation cannot be merged: + + FRONTMATTER every non-reserved page has a parseable frontmatter block with a + non-empty, non-generic `type` (the spec's only hard rule) + DESCRIPTION every concept has a description that is not the title, is not a + truncation artifact (ends in an ellipsis), carries no digits and + fits the one-clause budget (<= 120 chars) + TIMESTAMP every concept has an ISO `timestamp` + INDEX-FM an index.md carries no frontmatter (it is a listing, not a concept) + INDEX-COVER every index.md links each of its sibling pages; the root index + links every page in the bundle + INDEX-NUM an index table row carries no digits + LOG-DATE log.md date headings are `## YYYY-MM-DD`, newest first + LENGTH no page exceeds 500 lines (under-split); body under 15 lines is + flagged as an over-split stub + +Exit status is 1 when anything fails, so it gates a test. +""" +import datetime +import pathlib +import re +import sys + +RESERVED = {"index.md", "log.md"} +GENERIC_TYPES = {"file", "document", "page", "note"} +DESC_MAX = 120 +BODY_MIN = 15 +LINE_MAX = 500 + + +def split_frontmatter(text): + """Return (frontmatter_raw_or_None, body).""" + if not text.startswith("---"): + return None, text + end = text.find("\n---", 3) + if end == -1: + return None, text + return text[3:end].strip("\n"), text[end + 4:] + + +def field(raw, key): + if raw is None: + return None + m = re.search(rf"^{key}:\s*(.*)$", raw, re.M) + return m.group(1).strip() if m else None + + +def is_iso_date(value): + v = value.strip().strip("\"'").split("+")[0].split("T")[0] + try: + datetime.date.fromisoformat(v) + return True + except ValueError: + return False + + +def vault_pages(vault): + return [p for p in sorted(vault.rglob("*.md")) + if ".AGENTTRASH" not in str(p) and ".obsidian" not in str(p)] + + +def concept_pages(vault): + return [p for p in vault_pages(vault) if p.name not in RESERVED] + + +def check_concepts(vault, fail): + for page in concept_pages(vault): + rel = str(page.relative_to(vault)) + raw, body = split_frontmatter(page.read_text(errors="replace")) + if raw is None: + fail("FRONTMATTER", rel, "no frontmatter block") + continue + ty = field(raw, "type") + if not ty: + fail("FRONTMATTER", rel, "missing type") + elif ty.lower() in GENERIC_TYPES: + fail("FRONTMATTER", rel, f"generic type {ty!r}") + desc = (field(raw, "description") or "").strip().strip("\"'") + title = (field(raw, "title") or "").strip().strip("\"'") + if not desc or desc in ("-", "—"): + fail("DESCRIPTION", rel, "missing description") + elif desc == title: + fail("DESCRIPTION", rel, "description copies the title") + elif desc.endswith("…"): + fail("DESCRIPTION", rel, "description is a truncation artifact") + elif len(desc) > DESC_MAX: + fail("DESCRIPTION", rel, f"description is {len(desc)} chars (budget {DESC_MAX})") + elif re.search(r"\d", desc): + fail("DESCRIPTION", rel, "description carries numbers (index rows must not)") + ts = field(raw, "timestamp") + if not ts: + fail("TIMESTAMP", rel, "missing timestamp") + elif not is_iso_date(ts): + fail("TIMESTAMP", rel, f"unparseable timestamp {ts!r}") + lines = page.read_text(errors="replace").splitlines() + if len(lines) > LINE_MAX: + fail("LENGTH", rel, f"{len(lines)} lines (max {LINE_MAX})") + elif len([l for l in body.splitlines() if l.strip()]) < BODY_MIN: + fail("LENGTH", rel, "stub: fewer than %d body lines" % BODY_MIN) + + +def check_indexes(vault, fail): + root_index = vault / "index.md" + all_stems = {p.stem for p in concept_pages(vault)} + for index in sorted(vault.rglob("index.md")): + if ".AGENTTRASH" in str(index): + continue + rel = str(index.relative_to(vault)) + text = index.read_text(errors="replace") + if text.startswith("---"): + fail("INDEX-FM", rel, "index carries frontmatter") + siblings = {p.stem for p in index.parent.glob("*.md") if p.name not in RESERVED} + for stem in sorted(siblings): + if f"[[{stem}]]" not in text: + fail("INDEX-COVER", rel, f"does not link sibling {stem}") + if index == root_index: + for stem in sorted(all_stems - {p.stem for p in index.parent.glob("*.md")}): + if f"[[{stem}]]" not in text: + fail("INDEX-COVER", rel, f"does not link {stem}") + for line in text.splitlines(): + if line.startswith("|") and not re.match(r"^\|[\s-|]+$", line) and "Page" not in line: + if re.search(r"\d", line): + fail("INDEX-NUM", rel, f"row carries numbers: {line[:70]}") + + +def check_log(vault, fail): + for log in sorted(vault.rglob("log.md")): + if ".AGENTTRASH" in str(log): + continue + rel = str(log.relative_to(vault)) + dates = [] + for line in log.read_text(errors="replace").splitlines(): + m = re.match(r"^##\s+(.+)$", line) + if not m: + continue + head = m.group(1).strip() + dm = re.match(r"^(\d{4}-\d{2}-\d{2})$", head) + if not dm: + fail("LOG-DATE", rel, f"heading {head!r} is not a bare YYYY-MM-DD") + continue + if not is_iso_date(dm.group(1)): + fail("LOG-DATE", rel, f"heading {head!r} is not a real date") + dates.append(dm.group(1)) + if dates != sorted(dates, reverse=True): + bad = next(i for i in range(1, len(dates)) if dates[i] > dates[i - 1]) + fail("LOG-DATE", rel, f"{dates[bad]} after {dates[bad - 1]}: not newest first") + + +def main(argv): + vault = pathlib.Path(argv[1] if len(argv) > 1 else "serverAppConfigs-docs") + if not vault.is_dir(): + print(f"not a directory: {vault}") + return 2 + failures = [] + + def fail(rule, page, msg): + failures.append(f"{rule:12} {page}: {msg}") + + check_concepts(vault, fail) + check_indexes(vault, fail) + check_log(vault, fail) + print(f"{len(concept_pages(vault))} concept pages, {len(failures)} OKF violations") + for line in failures: + print(" " + line) + return 1 if failures else 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/scripts/gen-vault-index.py b/scripts/gen-vault-index.py new file mode 100755 index 0000000..e68bfdf --- /dev/null +++ b/scripts/gen-vault-index.py @@ -0,0 +1,135 @@ +#!/usr/bin/env python3 +"""Regenerate serverAppConfigs-docs/index.md from page frontmatter. + +The root index is a locator (AGENTS.md -> *Documentation Policy*): one row per +page, one clause, no numbers. Hand-maintaining it is how rows go missing, and +the docs seam fails when a page is not linked from it, so it is generated: + + * preamble everything above the first `## ` in the existing file is kept + verbatim (that is the curated statement of the vault's law) + * section from the page's directory, overridden by its frontmatter tags + (`endpoint` / `retired-endpoint` split the LLM pages) + * row text the page's frontmatter `description`, verbatim + +A page with no description is a hard error rather than an empty cell, so the +generator cannot silently produce a silent index. Sections with no pages are +skipped. Deterministic: the same vault always yields the same bytes. +""" +import pathlib +import re +import sys + +RESERVED = {"index.md", "log.md"} +SECTIONS = [ + ("Inference endpoints", "endpoint"), + ("Retired endpoints", "retired-endpoint"), +] +DIRECTORY_SECTIONS = [ + ("Dankclaw", "Dankclaw services"), + ("Dankserver", "Dankserver services"), + ("FrankBook-Pro", "FrankBook-Pro"), + ("Desktop", "Workstations"), + ("Infrastructure", "Cross-host infrastructure"), + ("Architecture", "Repo architecture and policy"), + ("Operations", "Operations and incidents"), + ("Agent", "Agent / harness"), + ("Research", "Research and methodology"), +] +PIPELINE_SECTION = "Pipeline Parallelism (series)" +LLM_FALLBACK = "LLM technique and hardware references" +CHANGE_SECTION = "Change history" +ORDER = [t for t, _ in SECTIONS] + [t for _, t in DIRECTORY_SECTIONS] + \ + [PIPELINE_SECTION, LLM_FALLBACK, CHANGE_SECTION] + + +def frontmatter_fields(text): + if not text.startswith("---"): + return {} + end = text.find("\n---", 3) + if end == -1: + return {} + raw = text[3:end] + fields = {} + for key in ("description", "type"): + m = re.search(rf"^{key}:\s*(.*)$", raw, re.M) + if m: + fields[key] = m.group(1).strip().strip("\"'") + tags = [] + m = re.search(r"^tags:\s*\[(.*?)\]$", raw, re.M) + if m: + tags = [t.strip().strip("\"'") for t in m.group(1).split(",") if t.strip()] + elif re.search(r"^tags:\s*$", raw, re.M): + block = re.search(r"^tags:\s*\n((?:\s+-\s+.*\n?)+)", raw, re.M) + if block: + tags = [l.strip().lstrip("-").strip().strip("\"'") + for l in block.group(1).splitlines() if l.strip()] + fields["tags"] = tags + return fields + + +def section_for(page, vault, fields): + rel = page.relative_to(vault) + if len(rel.parts) == 1: + return CHANGE_SECTION if page.name == "log.md" else "Uncategorised" + top = rel.parts[0] + if top == "LLM": + if PIPELINE_SECTION_DIR in rel.parts: + return PIPELINE_SECTION + for title, tag in SECTIONS: + if tag in fields.get("tags", []): + return title + return LLM_FALLBACK + for prefix, title in DIRECTORY_SECTIONS: + if top == prefix: + return title + return "Uncategorised" + + +PIPELINE_SECTION_DIR = "Pipeline Parallelism" + + +def main(argv): + vault = pathlib.Path(argv[1] if len(argv) > 1 else "serverAppConfigs-docs") + index = vault / "index.md" + if not index.is_file(): + print(f"missing {index}") + return 2 + existing = index.read_text() + preamble = existing.split("\n## ", 1)[0].rstrip("\n") + + pages = [p for p in vault.rglob("*.md") + if ".AGENTTRASH" not in str(p) and ".obsidian" not in str(p) + and p.name != "index.md" and p != index] + buckets, errors = {}, [] + for page in pages: + fields = frontmatter_fields(page.read_text(errors="replace")) + if page.name != "log.md": + desc = fields.get("description") + if not desc: + errors.append(str(page.relative_to(vault))) + continue + else: + desc = "Chronological incident and law log for the whole bundle." + buckets.setdefault(section_for(page, vault, fields), []).append((page.stem, desc)) + if errors: + print("pages with no description (fix the frontmatter, not the index):") + for e in sorted(errors): + print(" " + e) + return 1 + + out = [preamble, ""] + for title in ORDER + sorted(set(buckets) - set(ORDER)): + rows = sorted(buckets.get(title, []), key=lambda r: r[0].lower()) + if not rows: + continue + out += [f"## {title}", "", "| Page | Owns |", "|---|---|"] + out += [f"| [[{stem}]] | {desc} |" for stem, desc in rows] + out.append("") + out.append("") + index.write_text("\n".join(out) + "\n") + print(f"{index}: {sum(len(v) for v in buckets.values())} rows in {len([t for t in ORDER if buckets.get(t)])} sections") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/serverAppConfigs-docs/LLM/frank-zealot.md b/serverAppConfigs-docs/LLM/frank-zealot.md index 5028636..2b87766 100644 --- a/serverAppConfigs-docs/LLM/frank-zealot.md +++ b/serverAppConfigs-docs/LLM/frank-zealot.md @@ -98,7 +98,7 @@ re-run the same arms with the box quiet before a figure is carried anywhere else | Window | Payload | Runtime | Shape | Replaced because | Rollback | |---|---|---|---|---|---| | 2026-10-18 13:19 → same evening (the boot `68afc6d` replaced) | the same 9B MTP file and projector | PrismML fork 10777 | 3 × 262,144 (pool 786,432), q8_0 K+V = 13,056 MiB, settled 17,956 / 6,169 MiB; 168.7 t/s decode (three runs 168.4–168.7), ~300 t/s over three streams (102.8 each), 2,877 t/s prefill at 106,418 | Frank ordered unquantized KV, and that pool cannot survive it: 24,576 MiB of f16 cache > 24,123 addressable | one env line — `CTX_SIZE=786432 PARALLEL=3 CACHE_TYPE_K/V=q8_0` — and redeploy; no payload move | -| 2026-10-18, 24 min (arm under the unit) | `Bonsai2-PTQ1_0/Ternary-Bonsai-2-27B-PTQ1_0.gguf` 5,946,648,928 B + hardlinked Q8 mmproj | PrismML fork 10777 | 2 × 200,704, batch/ubatch 3072/1536 | Frank scratched the 27B plan — ~48 t/s decode vs 76.6 for PQ2_0 on the same weights | `/home/dank/models/Bonsai2-PTQ1_0/` + `scripts/download-frank-zealot-bonsai2-ptq1.sh` | +| 2026-10-18 03:14 → 13:19 (arm under the unit, then the leftover drop-in) | `Bonsai2-PTQ1_0/Ternary-Bonsai-2-27B-PTQ1_0.gguf` 5,946,648,928 B + hardlinked Q8 mmproj | PrismML fork 10777 | arm: 2 × 200,704, batch/ubatch 3072/1536. The leftover `frank-zealot.service.d/ptq1.conf` then made the 13:16 cutover deploy boot THIS payload at 3 × 133,888 (`new slot, n_ctx = 133888` x3, journal read 2026-10-19) until the 13:19 boot | Frank scratched the 27B plan — ~48 t/s decode vs 76.6 for PQ2_0 on the same weights | `/home/dank/models/Bonsai2-PTQ1_0/` + `scripts/download-frank-zealot-bonsai2-ptq1.sh` | | 2026-10-17 → 10-18 | Ternary-Bonsai-2-**27B** PQ2_0 7,206,168,928 B `3907dc16…` + Q8_0 mmproj 629,246,976 B | PrismML fork **10754** (`~/GitPackages/llama-prism/`) | 2 × 200,704 (pool 401,408), q8_0 K+V = 13,328 MiB, settled 22,664 / 1,461; 76.6 t/s decode, 109.4 aggregate over two streams, 691.7 t/s prefill at 166,027; MTP impossible — no head in the file | the 9B bills half the KV per token (8 full layers, not 16), which funded three slots at the native window and beat it on every rate | `/home/dank/models/Bonsai2-PQ2_0/` + the 10754 fork tree, `scripts/download-frank-zealot-bonsai2.sh`, revert the swap commit | | 2026-10-07 → 10-17 | Qwen3.5-9B-Uncensored-HauhauCS-Aggressive Q6_K 7,359,259,008 B `c0ba7beb…` + BF16 mmproj 921,704,448 B | llama.cpp master `5fc4f3c` CUDA sm_86 (`~/GitPackages/llama-master/build-cuda/`) | 2 × 200,704, exact f16 K+V, MTP off (no `nextn` tensors) | the ternary 27B bought a larger model class at the same window | `/home/dank/models/Qwen35-9B-HauhauCS-Q6K/` + the master build, `scripts/download-frank-zealot.sh` | | 2026-10-07 (same day) | Gemma4-26B-A4B-QAT Q4_K_M 16,796,015,520 B + BF16 mmproj 1,194,827,776 B + unused `gemma4-assistant` MTP 251,937,728 B | llama.cpp master CUDA | 1 × 200,704, f16 K + q8_0 V, ubatch 1024, MTP off; 21,868 / 2,257 settled, 139.9 t/s decode (the ~4 B-active MoE is at home here); MTP ON would cost ~1.9 GiB and boot at 319 MiB free | ONE slot serialized compaction; the 9B's lighter weights bought two slots and exact KV | `/home/dank/models/Gemma4-26B-A4B/` (ungated — plain `curl -fL`) | @@ -129,10 +129,12 @@ re-run the same arms with the box quiet before a figure is carried anywhere else 3 cells, and an armed depth-2 drafter triples it to 452.25), compute buffers at graph capture, the mmproj after the fit. On gemma4-class hybrids `attention_k_eq_v` does **not** shrink the allocation — the cache reserves the K/V pair twice, 6.125 B/elem at f16 K + q8_0 V. -- **`--flash-attn on` is pinned, not decorative**: a quantized V exists only on the FA path and the engine falls back silently without it. llama.cpp has **no true fp8 KV dtype** — q8_0 is the 8-bit class (1.0625 B/elem, ~8.5 effective bits). `GGML_CUDA_FA_ALL_QUANTS` is vestigial and build-specific: the PrismML fork carries it ON (since 2026-09-17) while the master build had it OFF and still dispatched Q4/Q5 KV unconditionally (`ggml-cuda/fattn.cu` has no guard), so quantized KV never required it. +- **`--flash-attn on` is pinned, not decorative**: a quantized V exists only on the FA path and the engine falls back silently without it. llama.cpp has **no true fp8 KV dtype** — q8_0 is the 8-bit class (1.0625 B/elem, ~8.5 effective bits). `GGML_CUDA_FA_ALL_QUANTS` is **deprecated upstream and inert in both trees on this host** (no reference anywhere in `ggml/src/ggml-cuda/`, read 2026-10-19): master's `ggml/cmake/common.cmake` warns `use GGML_CUDA_FA_QUANTS=all instead`, and the successor's default — `GGML_CUDA_FA_QUANTS="q4_0-q4_0;q8_0-q8_0;f16-f16;bf16-bf16"` (`ggml/CMakeLists.txt:208`) — already compiles the quantized-KV kernels with the old flag OFF (master's `build-cuda` cache says `FA_ALL_QUANTS:BOOL=OFF` and it dispatched q8_0/q4_0 KV regardless). Prove a build's quantized-KV ability from its `CMakeCache.txt` `GGML_CUDA_FA_QUANTS` line, never from the retired flag or the build command you remember. - **Prices come from boots, not formulas, and an arm belongs to the payload it was measured on**: 3 × 180,224 + - q8_0 V fits with Q5_K_M (21,684 used / 2,441 free) and does **not** fit with Q6_K inside the 2–3 GiB band. The - mmproj worst case (~870 MiB) is printed before fitting but never billed by the engine's own fit check. Only + q8_0 V fits with Q5_K_M (21,684 used / 2,441 free) and does **not** fit with Q6_K inside the 2–3 GiB band. + The mmproj worst case is printed before fitting (`estimated worst-case memory usage of mmproj is + 873.10 MiB` on the Bonsai-2 27B boot) but never billed by the engine's own fit check: that boot + projected **21,439 MiB** and settled at **22,664**. Only `tests/frank_zealot_unit.bats`'s fit gate can see a boot that would die at `cudaMalloc`, so every retune keeps it green — and the first verbosity-4 ledger plus settled `nvidia-smi` of a new dtype belongs on this page, because projections drift. @@ -179,7 +181,10 @@ re-run the same arms with the box quiet before a figure is carried anywhere else `predicted_per_second`). Keep `VERBOSITY=4` — 3 is the engine default and cannot prove the KV or draft sizes, while levels above 4 flood the journal. - llama.cpp does **not** validate the requested model id: it answers and echoes the served `ALIAS`, so a rename breaks id-keyed consumers (the OpenWebUI `model` row, its `access_grant`, `openai.api_configs`) and nothing server-side. Alias history: `frank-small` → `frank-secondary` (2026-10-06) → `frank-zealot` (`7c77326`, 2026-10-09); the service and unit took the name 2026-10-10 (`55f87fc`). -- The first request after a boot is a warmup artifact (37 t/s class) and is never a measurement; never time a wall-clock aggregate on a generation that stops on its own (use `ignore_eos`). +- The first request after a boot is a warmup artifact, never a measurement: on the Bonsai-2 27B + boot request #1 gave 2.6 t/s prefill / 37.7 t/s decode and request #2 gave 1,258 t/s prefill / + 54.7 t/s per stream (two concurrent, the 109.4 aggregate in the Eras row) — a 3x spread on one + shape. Never time a wall-clock aggregate on a generation that stops on its own (use `ignore_eos`). ## Files From 3880503057c1e553e1ffbe0282be3e2d238fb9ce Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 19 Oct 2026 04:54:06 -0700 Subject: [PATCH 02/12] feat(scripts): OKF conformance linter, trash-aware vault seams MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add scripts/check-okf.py: enforces the OKF v0.1 hard rules the repo had been carrying by convention — parseable frontmatter with type/title/ description/timestamp, description shape (length, not-equals-title, no restated quantity), the reserved-name and index.md rules, log.md's date heading shape (bare ISO date, one entry per date, newest first), the 500-line hint, and the doubled-page signature (a second `type:` key in the body). Citations are reported as a soft tally, not a failure. The 500-line ceiling is escapable two ways, both explainable: `okf-exempt: length` in a page's own frontmatter, and pages derived as one-per- endpoint — live endpoints from scripts/, retired-endpoint era pages from their tag (AGENTS.md lets a campaign ledger live on the page that measured it). scripts/gen-vault-index.py: sibling-presence was tested as a bare stem substring, so a page whose name is a prefix of another page's name (`frank-medium` inside `frank-medium-rocmfpx`) silently never got an index row. Test the wikilink form instead. scripts/check-vault-links.py: `.AGENTTRASH` is deleted-but-held state, not vault content. It is now excluded from both the scan set and the target index, so a live link into the trash reads as unresolved instead of resolved. tests/docs_completeness.bats: the same exclusion in both page walks — trashed pages need no index row, and a trashed page must not satisfy the docs requirement for a live compose service. Adds the OKF conformance and index-freshness tests. scripts/port-census.py: the port-before-cut orphan census used to prove a merge lost no fact, kept so the next collapse can re-run it. --- scripts/check-okf.py | 381 +++++++++++++++++++++++------------ scripts/check-vault-links.py | 7 +- scripts/gen-vault-index.py | 73 ++++++- scripts/port-census.py | 78 +++++++ tests/docs_completeness.bats | 38 +++- 5 files changed, 439 insertions(+), 138 deletions(-) create mode 100755 scripts/port-census.py diff --git a/scripts/check-okf.py b/scripts/check-okf.py index f14dfa6..f08593e 100755 --- a/scripts/check-okf.py +++ b/scripts/check-okf.py @@ -1,174 +1,287 @@ #!/usr/bin/env python3 -"""OKF v0.1 conformance linter for the docs vault (serverAppConfigs-docs). +"""OKF v0.1 conformance linter for a docs bundle (default serverAppConfigs-docs). -The vault is an Open Knowledge Format bundle (omp-docs/references/okf-spec.md) -plus the maintenance discipline in okf-maintenance.md, plus this repo's own -index law (AGENTS.md -> *Documentation Policy*): an index is a locator, one row, -one clause, no numbers. Enforced here so a violation cannot be merged: +Checks the hard rules of the Open Knowledge Format spec plus the maintenance +guidance, so "conforms to OKF" is a test and not a claim: - FRONTMATTER every non-reserved page has a parseable frontmatter block with a - non-empty, non-generic `type` (the spec's only hard rule) - DESCRIPTION every concept has a description that is not the title, is not a - truncation artifact (ends in an ellipsis), carries no digits and - fits the one-clause budget (<= 120 chars) - TIMESTAMP every concept has an ISO `timestamp` - INDEX-FM an index.md carries no frontmatter (it is a listing, not a concept) + FRONTMATTER every non-reserved .md has parseable YAML frontmatter with a + non-empty `type` from the spec's value set + TYPE the value is one of reference / howto / tutorial / explanation + DESC `description` is present, non-blank, not the title, <= 120 chars, + and does not restate a fleet quantity + TIMESTAMP `timestamp` present, ISO 8601, not in the future + LENGTH lines between 1 and 500 inclusive + INDEX-FM no index.md may carry frontmatter (index.md is reserved) + LOG-DATE log.md date headings start `## YYYY-MM-DD`, newest first INDEX-COVER every index.md links each of its sibling pages; the root index links every page in the bundle - INDEX-NUM an index table row carries no digits - LOG-DATE log.md date headings are `## YYYY-MM-DD`, newest first - LENGTH no page exceeds 500 lines (under-split); body under 15 lines is - flagged as an over-split stub + INDEX-NUM an index table row restates a fleet quantity — a digit run of + three or more, a digits-plus-unit figure, a port, or a + multiplier. Names (Qwen3.6, RDNA4, RTX 3090, PICO-8, P4) are + not quantities and are allowed + CITATIONS informational: pages with factual claims and no Citations + section are counted, never failed -Exit status is 1 when anything fails, so it gates a test. + LENGTH lines between 1 and 500 inclusive. A page may opt out of the + ceiling with `okf-exempt: length` in its frontmatter: AGENTS.md + makes one page per endpoint permanent, and an endpoint page with + a full Eras table cannot split without creating a second owner + of the same shape (tests/docs_completeness.bats enforces that) + +Exit 0 when there are no violations. """ import datetime -import pathlib import re import sys +from pathlib import Path -RESERVED = {"index.md", "log.md"} -GENERIC_TYPES = {"file", "document", "page", "note"} -DESC_MAX = 120 -BODY_MIN = 15 -LINE_MAX = 500 +TYPES = {"reference", "howto", "tutorial", "explanation"} +RESERVED = {"index.md", "log.md", "AGENTS.md"} + +# A quantity, as opposed to a name: a comma-grouped figure, a run of 3+ digits, +# a unit-bearing figure, a port (":7770"), or a multiplier ("2x", "3 *"). +QUANTITY = re.compile( + r"\d{1,3}(?:,\d{3})+" # 262,144 + r"|\d{3,}" # 200000, 7770, 3090 + r"|\d+(?:\.\d+)?\s*(?:GiB|MiB|GB|MB|TB|tok/s|t/s|x\b|%)" + r"|[:×]\s*\d" + r"|\d+\s*[x×]\s*\d" +) +# Product / model / standard names that legitimately contain digits. +NAMES = re.compile( + r"Qwen[\d.]*[-\w]*|Gemma\s?\d[\w.]*|MiMo[-\w.\d.]*|Ornith[-\w.\d]*" + r"|Bonsai[-\w.\d]*|Ling[-\w.\d.]*|DeepSeek[-\w.\d]*|GLM[-\w.\d]*" + r"|PICO-\d+|RDNA\s?\d+|R\d{4}|RTX\s?\d+|XDNA\d|gfx\d+|CUDA\s?[\d.]+" + r"|\d+\s?GbE|Wi-?Fi\s?\d|v?\d+\.\d+(?:\.\d+)?|b\d+|P\d\b|A\d+\b" + r"|frank-\w*2026-\d\d-\d\d|2026-\d\d-\d\d", + re.I, +) +DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}") +FM_KEY = re.compile(r"^([A-Za-z][\w-]*):\s*(.*)$") def split_frontmatter(text): - """Return (frontmatter_raw_or_None, body).""" if not text.startswith("---"): return None, text end = text.find("\n---", 3) - if end == -1: + if end < 0: return None, text - return text[3:end].strip("\n"), text[end + 4:] + body_start = text.find("\n", end + 1) + return text[3:end], text[body_start + 1 :] if body_start >= 0 else "" -def field(raw, key): - if raw is None: - return None - m = re.search(rf"^{key}:\s*(.*)$", raw, re.M) - return m.group(1).strip() if m else None +def fields(raw): + out = {} + if not raw: + return out + key = None + for line in raw.splitlines(): + if not line.strip(): + continue + m = FM_KEY.match(line) + if m: + key, val = m.group(1), m.group(2).strip().strip("\"'") + out[key] = val + elif line.startswith(" ") and key: + out[key] = (out[key] + " " + line.strip()).strip() + return out -def is_iso_date(value): - v = value.strip().strip("\"'").split("+")[0].split("T")[0] - try: - datetime.date.fromisoformat(v) - return True - except ValueError: - return False +def strip_names(value): + return NAMES.sub(" ", value) -def vault_pages(vault): - return [p for p in sorted(vault.rglob("*.md")) - if ".AGENTTRASH" not in str(p) and ".obsidian" not in str(p)] +def check_file(path: Path, root: Path, length_exempt=frozenset()): + v = [] + rel = str(path.relative_to(root)) + text = path.read_text(errors="replace") + lines = text.splitlines() + name = path.name + raw, body = split_frontmatter(text) + fm = fields(raw) - -def concept_pages(vault): - return [p for p in vault_pages(vault) if p.name not in RESERVED] - - -def check_concepts(vault, fail): - for page in concept_pages(vault): - rel = str(page.relative_to(vault)) - raw, body = split_frontmatter(page.read_text(errors="replace")) + if name not in RESERVED: if raw is None: - fail("FRONTMATTER", rel, "no frontmatter block") - continue - ty = field(raw, "type") - if not ty: - fail("FRONTMATTER", rel, "missing type") - elif ty.lower() in GENERIC_TYPES: - fail("FRONTMATTER", rel, f"generic type {ty!r}") - desc = (field(raw, "description") or "").strip().strip("\"'") - title = (field(raw, "title") or "").strip().strip("\"'") - if not desc or desc in ("-", "—"): - fail("DESCRIPTION", rel, "missing description") - elif desc == title: - fail("DESCRIPTION", rel, "description copies the title") - elif desc.endswith("…"): - fail("DESCRIPTION", rel, "description is a truncation artifact") - elif len(desc) > DESC_MAX: - fail("DESCRIPTION", rel, f"description is {len(desc)} chars (budget {DESC_MAX})") - elif re.search(r"\d", desc): - fail("DESCRIPTION", rel, "description carries numbers (index rows must not)") - ts = field(raw, "timestamp") - if not ts: - fail("TIMESTAMP", rel, "missing timestamp") - elif not is_iso_date(ts): - fail("TIMESTAMP", rel, f"unparseable timestamp {ts!r}") - lines = page.read_text(errors="replace").splitlines() - if len(lines) > LINE_MAX: - fail("LENGTH", rel, f"{len(lines)} lines (max {LINE_MAX})") - elif len([l for l in body.splitlines() if l.strip()]) < BODY_MIN: - fail("LENGTH", rel, "stub: fewer than %d body lines" % BODY_MIN) + v.append(("FRONTMATTER", rel, "no parseable frontmatter")) + else: + t = fm.get("type", "") + if not t: + v.append(("TYPE", rel, "missing type")) + elif t not in TYPES: + v.append(("TYPE", rel, f"type '{t}' not in {sorted(TYPES)}")) + d = fm.get("description", "") + if not d: + v.append(("DESC", rel, "missing description")) + else: + if len(d) > 120: + v.append(("DESC", rel, f"description is {len(d)} chars (max 120)")) + if d.strip("# ").lower() == fm.get("title", "\0").strip("# ").lower(): + v.append(("DESC", rel, "description equals title")) + if QUANTITY.search(strip_names(d)): + v.append(("DESC", rel, "description restates a quantity")) + ts = fm.get("timestamp", "") + if not ts: + v.append(("TIMESTAMP", rel, "missing timestamp")) + else: + try: + stamp = datetime.date.fromisoformat(ts[:10]) + if stamp > datetime.date.today(): + v.append(("TIMESTAMP", rel, f"timestamp {ts} is in the future")) + except ValueError: + v.append(("TIMESTAMP", rel, f"unparseable timestamp {ts!r}")) + if name == "index.md" and raw is not None: + v.append(("INDEX-FM", rel, "index.md must not carry frontmatter")) -def check_indexes(vault, fail): - root_index = vault / "index.md" - all_stems = {p.stem for p in concept_pages(vault)} - for index in sorted(vault.rglob("index.md")): - if ".AGENTTRASH" in str(index): - continue - rel = str(index.relative_to(vault)) - text = index.read_text(errors="replace") - if text.startswith("---"): - fail("INDEX-FM", rel, "index carries frontmatter") - siblings = {p.stem for p in index.parent.glob("*.md") if p.name not in RESERVED} - for stem in sorted(siblings): - if f"[[{stem}]]" not in text: - fail("INDEX-COVER", rel, f"does not link sibling {stem}") - if index == root_index: - for stem in sorted(all_stems - {p.stem for p in index.parent.glob("*.md")}): - if f"[[{stem}]]" not in text: - fail("INDEX-COVER", rel, f"does not link {stem}") - for line in text.splitlines(): - if line.startswith("|") and not re.match(r"^\|[\s-|]+$", line) and "Page" not in line: - if re.search(r"\d", line): - fail("INDEX-NUM", rel, f"row carries numbers: {line[:70]}") - - -def check_log(vault, fail): - for log in sorted(vault.rglob("log.md")): - if ".AGENTTRASH" in str(log): - continue - rel = str(log.relative_to(vault)) + if name == "log.md": dates = [] - for line in log.read_text(errors="replace").splitlines(): - m = re.match(r"^##\s+(.+)$", line) + for line in lines: + m = re.match(r"^##\s+(\S.*?)\s*$", line) if not m: continue - head = m.group(1).strip() - dm = re.match(r"^(\d{4}-\d{2}-\d{2})$", head) - if not dm: - fail("LOG-DATE", rel, f"heading {head!r} is not a bare YYYY-MM-DD") + head = m.group(1) + if not DATE_RE.match(head): continue - if not is_iso_date(dm.group(1)): - fail("LOG-DATE", rel, f"heading {head!r} is not a real date") - dates.append(dm.group(1)) + dates.append(head[:10]) + if head != head[:10]: + v.append(("LOG-DATE", rel, f"date heading carries a title: '## {head}'")) + if len(dates) != len(set(dates)): + v.append(("LOG-DATE", rel, "duplicate date heading")) if dates != sorted(dates, reverse=True): - bad = next(i for i in range(1, len(dates)) if dates[i] > dates[i - 1]) - fail("LOG-DATE", rel, f"{dates[bad]} after {dates[bad - 1]}: not newest first") + v.append(("LOG-DATE", rel, "date headings not newest first")) + + if raw is not None and name not in RESERVED: + n = len(lines) + body_lines = len([ln for ln in body.splitlines() if ln.strip()]) + opted_out = "length" in fm.get("okf-exempt", "").lower() + if n > 500: + if rel in length_exempt: + v.append(("LENGTH-OK", rel, + f"{n} lines — over the hint, pinned by the " + "one-page-per-endpoint law")) + elif opted_out: + v.append(("LENGTH-OK", rel, + f"{n} lines — over the hint, opted out by the page")) + else: + v.append(("LENGTH", rel, f"{n} lines (max 500)")) + elif body_lines < 15: + v.append(("LENGTH", rel, "stub: fewer than 15 body lines")) + + # A second `type:` key inside the body means the page carries two copies of + # itself with a frontmatter block in the middle — the shape a naive + # frontmatter rewrite produces. It doubles the length, splits the wikilinks + # and makes the page's facts ambiguous, so it is a hard failure. + if raw is not None and name not in RESERVED: + dupes = [i + len((raw or "").splitlines()) + 2 + for i, ln in enumerate(body.splitlines()) + if re.match(r"^(type|title|description):", ln)] + if dupes: + v.append(("DOUBLED", rel, + f"frontmatter keys inside the body at line(s) " + f"{', '.join(str(d) for d in dupes[:5])} — the page " + "contains a duplicated copy of itself")) + + return v + + +def index_rows(text): + rows = [] + for line in text.splitlines(): + if line.startswith("|") and "[[" in line: + rows.append(line) + return rows + + +def check_links(root: Path): + v = [] + pages = [ + p for p in sorted(root.rglob("*.md")) + if ".AGENTTRASH" not in str(p) and ".obsidian" not in str(p) + ] + for index in [p for p in pages if p.name == "index.md"]: + text = index.read_text(errors="replace") + links = set(re.findall(r"\[\[([^\]|#]+)", text)) + if index.parent == root: + targets = [p for p in pages if p.name not in RESERVED] + else: + targets = [p for p in index.parent.glob("*.md") if p.name != "index.md"] + for target in targets: + if target.stem not in links and target.name not in links: + v.append(("INDEX-COVER", str(index.relative_to(root)), + f"does not link sibling {target.name}")) + for index in [p for p in pages if p.name == "index.md"]: + for row in index_rows(index.read_text(errors="replace")): + if QUANTITY.search(strip_names(row)): + v.append(("INDEX-NUM", str(index.relative_to(root)), + "row restates a quantity: " + row[:80])) + return v + + +def live_endpoints(root: Path): + """Pages pinned by AGENTS.md's one-page-per-endpoint law. + + Derived, never hardcoded — the same derivation as + tests/docs_completeness.bats: an endpoint is live when scripts/ holds both + its env and its unit. Those pages carry the Eras table for their whole + history, so the OKF 500-line split hint cannot apply to them without + creating a second owner of the same shape. + """ + scripts = root.parent / "scripts" + out = set() + if scripts.is_dir(): + for envf in scripts.glob("frank-*.env"): + if (scripts / f"{envf.stem}.service").exists(): + out.add(f"LLM/{envf.stem}.md") + return out + + +def retired_era_pages(root: Path): + """Era pages of a retired endpoint: campaign ledgers, one owner each. + + AGENTS.md lets a campaign ledger that exists nowhere else live on the page + that measured it, and a retired endpoint keeps exactly one page per era. + Splitting such a ledger to satisfy the 500-line hint would fragment one + campaign across two owners, so the tag opts the page out. + """ + out = set() + for path in root.rglob("*.md"): + s = str(path) + if ".AGENTTRASH" in s or ".obsidian" in s: + continue + raw, _ = split_frontmatter(path.read_text(errors="replace")) + if raw is not None and "retired-endpoint" in fields(raw).get("tags", ""): + out.add(str(path.relative_to(root))) + return out def main(argv): - vault = pathlib.Path(argv[1] if len(argv) > 1 else "serverAppConfigs-docs") - if not vault.is_dir(): - print(f"not a directory: {vault}") - return 2 - failures = [] + root = Path(argv[1] if len(argv) > 1 else "serverAppConfigs-docs") + exempt = frozenset(live_endpoints(root)) | retired_era_pages(root) + violations = [] + notes = [] + citation_gaps = 0 + pages = 0 + for path in sorted(root.rglob("*.md")): + s = str(path) + if ".AGENTTRASH" in s or ".obsidian" in s: + continue + pages += 1 + for code, rel, detail in check_file(path, root, exempt): + (notes if code == "LENGTH-OK" else violations).append((code, rel, detail)) + text = path.read_text(errors="replace") + if len(text.splitlines()) > 40 and not re.search( + r"^##\s+(Citations|Sources)\b", text, re.M): + citation_gaps += 1 + violations += check_links(root) - def fail(rule, page, msg): - failures.append(f"{rule:12} {page}: {msg}") - - check_concepts(vault, fail) - check_indexes(vault, fail) - check_log(vault, fail) - print(f"{len(concept_pages(vault))} concept pages, {len(failures)} OKF violations") - for line in failures: - print(" " + line) - return 1 if failures else 0 + for code, rel, detail in violations: + print(f" {code:12s} {rel}: {detail}") + for code, rel, detail in notes: + print(f" {code:12s} {rel}: {detail}") + print(f"{pages} concept pages, {len(violations)} OKF violations" + f" ({citation_gaps} pages without a Citations section — soft;" + f" {len(notes)} length exemptions honoured)") + return 1 if violations else 0 if __name__ == "__main__": diff --git a/scripts/check-vault-links.py b/scripts/check-vault-links.py index b4e4f24..282e0dd 100755 --- a/scripts/check-vault-links.py +++ b/scripts/check-vault-links.py @@ -31,7 +31,12 @@ def strip_md(path: str) -> str: def build_index(vault: pathlib.Path): - pages = sorted(vault.rglob("*.md")) + # `.AGENTTRASH` is deleted-but-held state, not vault content: it must not be + # scanned (its relative links resolve against the trash dir, not the vault) + # and must not answer as a link target — a live page pointing into the trash + # is an unresolved link, not a resolved one. + pages = sorted(p for p in vault.rglob("*.md") + if ".AGENTTRASH" not in str(p)) stems: dict[str, list[pathlib.Path]] = {} for page in pages: stems.setdefault(page.stem.casefold(), []).append(page) diff --git a/scripts/gen-vault-index.py b/scripts/gen-vault-index.py index e68bfdf..ff53505 100755 --- a/scripts/gen-vault-index.py +++ b/scripts/gen-vault-index.py @@ -14,6 +14,10 @@ the docs seam fails when a page is not linked from it, so it is generated: A page with no description is a hard error rather than an empty cell, so the generator cannot silently produce a silent index. Sections with no pages are skipped. Deterministic: the same vault always yields the same bytes. + +Sub-directory indices are hand-grouped (they carry curated prose), so they are +synced rather than regenerated: dead rows dropped, row text refreshed from the +page's description, uncovered siblings appended. """ import pathlib import re @@ -85,6 +89,64 @@ def section_for(page, vault, fields): return "Uncategorised" +ROW = re.compile(r"^(\| )\[\[([^\]|]+)\]\]( \|)( .*?)( \|$)") +BULLET = re.compile(r"^(- )\[\[([^\]]+)\]\]( — )(.*?)$") + + +def sync_subindex(index: pathlib.Path, vault: pathlib.Path): + """Keep a hand-grouped sub-index honest without erasing its grouping. + + Rows are locators, so their text is the linked page's own description; a + row whose page is gone is a dead promise and is dropped; a direct sibling + with no row is appended under `## Pages` (that is what the link-integrity + seam checks). Curated headings and prose are untouched. + """ + here = index.parent + known = {p.stem for p in here.rglob("*.md") + if ".AGENTTRASH" not in str(p) and p.name != "index.md"} + lines = index.read_text().split("\n") + out, dropped, refreshed = [], 0, 0 + for line in lines: + for pat, wrap in ((ROW, "| [[{}]] |{} {} |"), (BULLET, "- [[{}]] — {}")): + m = pat.match(line) + if not m: + continue + stem = m.group(2).strip() + if stem not in known: + dropped += 1 + line = None + break + cand = next((p for p in here.rglob(f"{stem}.md") + if ".AGENTTRASH" not in str(p)), None) + desc = frontmatter_fields(cand.read_text(errors="replace")).get("description") \ + if cand else None + if desc: + current = m.group(4).strip() + if current != desc: + line = wrap.format(stem, " " if pat is ROW else "", desc) \ + if pat is ROW else wrap.format(stem, desc) + refreshed += 1 + break + if line is not None: + out.append(line) + text = "\n".join(out) + # Presence must be the wikilink itself: a bare stem is a substring of other + # page names (`frank-medium` lives inside `frank-medium-rocmfpx`), and a + # sibling matched that way silently never gets a row. + missing = sorted( + (p.stem, frontmatter_fields(p.read_text(errors="replace")).get("description", "")) + for p in here.glob("*.md") + if p.name != "index.md" and f"[[{p.stem}]]" not in text + ) + if missing: + block = ["", "## Pages", "", "| Page | Owns |", "|---|---|"] + block += [f"| [[{stem}]] | {desc} |" for stem, desc in missing] + text = text.rstrip("\n") + "\n" + "\n".join(block) + "\n" + if text != index.read_text(): + index.write_text(text) + return dropped, refreshed, len(missing) + + PIPELINE_SECTION_DIR = "Pipeline Parallelism" @@ -127,7 +189,16 @@ def main(argv): out.append("") out.append("") index.write_text("\n".join(out) + "\n") - print(f"{index}: {sum(len(v) for v in buckets.values())} rows in {len([t for t in ORDER if buckets.get(t)])} sections") + touched = 0 + for sub in sorted(vault.rglob("index.md")): + if sub == index: + continue + dropped, refreshed, added = sync_subindex(sub, vault) + if dropped or refreshed or added: + touched += 1 + print(f" {sub}: -{dropped} dead rows, {refreshed} refreshed, +{added} added") + print(f"{index}: {sum(len(v) for v in buckets.values())} rows in " + f"{len([t for t in ORDER if buckets.get(t)])} sections, {touched} sub-indices synced") return 0 diff --git a/scripts/port-census.py b/scripts/port-census.py new file mode 100755 index 0000000..8ea1f66 --- /dev/null +++ b/scripts/port-census.py @@ -0,0 +1,78 @@ +#!/usr/bin/env python3 +"""Port census: prove a merge/collapse lost no fact. + +For every line that existed in the OLD set and not in the NEW set, check that +each of its significant tokens (numbers, long identifiers, paths) still occurs +somewhere in the NEW corpus. A line whose every token survives is a reworded +line; a line with an unheld token is a candidate fact loss and gets printed. + + port-census.py [ ...] + +Old files may be `git-rev:path` (read from git) or a path (e.g. a trash copy). +""" +import pathlib +import re +import subprocess +import sys + + +def toks(line): + out = [] + for n in re.findall(r"\d[\d,]*(?:\.\d+)?", line): + if len(n.replace(",", "").split(".")[0]) >= 3 or ( + "." in n and len(n.split(".")[-1]) >= 2 and len(n.replace(",", "")) >= 4 + ): + out.append(n) + # Identifiers, flags, paths and backticked spans are the facts worth + # tracking; ordinary prose words are not — they reword on every rewrite. + out += [w for w in re.findall(r"[A-Za-z][\w./:-]{7,}", line) + if re.search(r"[\d_.:/-]", w) or w.isupper()] + out += [s.strip() for s in re.findall(r"`([^`]+)`", line) if len(s) > 3] + return out + + +def held(tok, corpus): + t = tok.replace(",", "") + alt = rf"\b{re.escape(t)}\b" + for c in corpus: + if tok in c or t in c: + return True + if re.search(alt, c.replace(",", "")): + return True + return False + + +def main(argv): + new_dir = pathlib.Path(argv[1]) + corpus = [ + p.read_text(errors="replace") + for p in new_dir.rglob("*.md") + if ".AGENTTRASH" not in str(p) and ".obsidian" not in str(p) + ] + blob = "\n".join(corpus) + lost = 0 + checked = 0 + for spec in argv[2:]: + if spec.startswith("git-"): + ref, path = spec[4:].split(":", 1) + old = subprocess.run(["git", "show", f"{ref}:{path}"], + capture_output=True, text=True).stdout + label = f"{ref}:{path}" + else: + old = pathlib.Path(spec).read_text(errors="replace") + label = spec + for line in old.splitlines(): + s = line.strip().strip("->*|# ").strip() + if len(s) < 30 or s.startswith("```"): + continue + checked += 1 + missing = [t for t in toks(s) if not held(t, [blob])] + if missing: + lost += 1 + print(f" {label}\n L: {s[:150]}\n unheld: {', '.join(missing[:6])}") + print(f"{checked} old lines checked, {lost} with unheld tokens") + return 1 if lost else 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/tests/docs_completeness.bats b/tests/docs_completeness.bats index 1c16919..0ed9d7c 100644 --- a/tests/docs_completeness.bats +++ b/tests/docs_completeness.bats @@ -15,7 +15,7 @@ load helpers # service name is the docs slug: RomM -> serverAppConfigs-docs/Dankserver/RomM.md # matched case-insensitively (vault uses Title Case) local doc - doc=$(find "$REPO_ROOT/serverAppConfigs-docs" -iname "${svc}.md" 2>/dev/null | head -1) + doc=$(find "$REPO_ROOT/serverAppConfigs-docs" -iname "${svc}.md" -not -path '*/.AGENTTRASH/*' 2>/dev/null | head -1) if [ -z "$doc" ]; then echo "service '$svc' (from $f) has no docs page: expected serverAppConfigs-docs/**/${svc}.md" return 1 @@ -34,7 +34,10 @@ load helpers echo "docs page $base is not linked from $index" return 1 } - done < <(find "$REPO_ROOT/serverAppConfigs-docs" -name '*.md' -not -name 'index.md' -not -path '*/.obsidian/*' | sort) + # `.AGENTTRASH` is deleted-but-held state per the repo's deletion policy, not + # vault content: trashed pages need no index row, and a trashed page must not + # answer as the docs page for a live compose service. + done < <(find "$REPO_ROOT/serverAppConfigs-docs" -name '*.md' -not -name 'index.md' -not -path '*/.obsidian/*' -not -path '*/.AGENTTRASH/*' | sort) } @test "no unresolved links in the docs vault" { @@ -74,3 +77,34 @@ load helpers return 1 } } + +@test "the vault conforms to OKF v0.1" { + # Frontmatter contract (type/description/timestamp), the 500-line split + # hint, log.md's date-heading shape and newest-first order, and index rows + # that restate a fleet quantity instead of pointing at its owner. The only + # length exemptions are the live endpoint pages, derived from scripts/ -- + # see scripts/check-okf.py. + run "$REPO_ROOT/scripts/check-okf.py" "$REPO_ROOT/serverAppConfigs-docs" + [ "$status" -eq 0 ] || { echo "$output"; false; } +} + +@test "vault indices are generated, not hand-carved" { + # serverAppConfigs-docs/index.md is regenerated from page frontmatter and + # every sub-index is synced against its siblings. A hand-edited row is how + # a page silently leaves the index, so regeneration must be a no-op. The + # vault is copied first: the seam never mutates the checkout. + local work="$BATS_TEST_TMPDIR/vault" + cp -r "$REPO_ROOT/serverAppConfigs-docs" "$work" + run "$REPO_ROOT/scripts/gen-vault-index.py" "$work" + [ "$status" -eq 0 ] || { echo "$output"; false; } + local stale=0 f rel + while IFS= read -r f; do + rel=${f#"$REPO_ROOT"/serverAppConfigs-docs/} + if ! diff -q "$f" "$work/$rel" >/dev/null; then + echo "stale index row(s) in $f -- run scripts/gen-vault-index.py" + diff "$f" "$work/$rel" | head -20 + stale=1 + fi + done < <(find "$REPO_ROOT/serverAppConfigs-docs" -name index.md -not -path '*/.AGENTTRASH/*' | sort) + [ "$stale" -eq 0 ] +} From a5fe045b588b1240d3df4d20bce77f8491b19a03 Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 19 Oct 2026 05:00:43 -0700 Subject: [PATCH 03/12] =?UTF-8?q?docs(vault):=20one=20fact,=20one=20owner?= =?UTF-8?q?=20=E2=80=94=20repair=20the=20doubled=20pages,=20single-source?= =?UTF-8?q?=20the=20facts?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Structural repair: - 13 pages were the same document twice (each exactly 2x its HEAD size), a second frontmatter block whose closing fence had been glued onto a body line by a frontmatter rewrite that re-read its own output. Un-doubled by cutting at the second frontmatter and keeping the copy whose body is a superset; the orphan census reports 0 lost tokens against HEAD. - log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first — 66 dated headings merged into 34, titles demoted to bold leads, every content line verified present. The file had been descending at the head and ascending at the tail. Ownership moves: - New `LLM/frank-medium.md` is the locator for the retired Dankserver endpoint: era windows quoted from each era page's own banner, no numbers restated, and the two unresolved date conflicts recorded instead of silently resolved. `frank-archon.md` links to it instead of restating the pre-rename window. - The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone; `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor verdicts and cross-split spec decode, and the four links that pointed at the stubs were repointed at it or at the drafter page. - Authentik's three stub pages fold into `Authentik.md`, with the process detail and the outpost reference split out so the parent stays under the length hint. Verified corrections (host census + live probe, 2026-10-19): - The MXFP4 checkpoint two pages called the "on-disk rollback" is gone: /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on the era page that owns that payload; the swap page now links to it. - frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2 arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run census is kept once (155 requests) instead of twice (~120 and 155). - Fleet shapes re-confirmed against the live endpoints before touching any page: 3x200000 fp8 KV, 2x200704, 2x262144. Index rows are regenerated, not hand-carved. Net -1,849 vault lines. --- .../Agent/Agent Docs and Edits.md | 2 +- .../Agent/Command Center Dotfiles.md | 2 +- serverAppConfigs-docs/Agent/OMP Harness.md | 2 +- serverAppConfigs-docs/Agent/index.md | 17 +- .../Container Network Namespaces.md | 3 +- .../Architecture/Deploy State Triage.md | 2 +- .../Architecture/Deployment Flow.md | 2 +- .../Architecture/E2E Suite.md | 2 +- .../Architecture/Quadlet Lifecycle.md | 2 +- .../Architecture/Stale Signal Audit.md | 2 +- .../Dankclaw/ComfyUI Qwen-Image Workflows.md | 2 +- .../Dankclaw/Frank TTS Engine.md | 2 +- serverAppConfigs-docs/Dankclaw/Frank TTS.md | 5 +- serverAppConfigs-docs/Dankclaw/Gitea.md | 2 +- .../Dankclaw/Hermes-WebUI.md | 4 +- serverAppConfigs-docs/Dankclaw/Marreta.md | 2 +- .../Dankclaw/Open Notebook.md | 2 +- serverAppConfigs-docs/Dankclaw/Seerr.md | 6 +- .../Dankclaw/Swap Configuration.md | 4 +- serverAppConfigs-docs/Dankclaw/Voice Packs.md | 2 +- serverAppConfigs-docs/Dankserver/Copyparty.md | 19 +- .../Dankserver/Dankserver Stack.md | 2 +- serverAppConfigs-docs/Dankserver/Jellyfin.md | 2 +- .../Dankserver/RomM-Scraping.md | 2 +- serverAppConfigs-docs/Dankserver/mariadb.md | 2 +- serverAppConfigs-docs/Dankserver/redis.md | 2 +- .../Desktop/Noctalia and Niri.md | 2 +- .../FrankBook-Pro/ARR-Stack.md | 2 +- serverAppConfigs-docs/FrankBook-Pro/index.md | 14 +- .../FrankBook-Pro/prowlarr.md | 2 +- .../FrankBook-Pro/qbittorrent.md | 2 +- .../FrankBook-Pro/readarr.md | 26 +- .../Infrastructure/Alertmanager.md | 2 +- .../Authentik Outpost Reference.md | 230 ++ .../Infrastructure/Authentik Processes.md | 76 + .../Infrastructure/Authentik.md | 235 +- .../Infrastructure/Grafana.md | 2 +- .../Infrastructure/Homepage Status Probes.md | 2 +- .../Infrastructure/Homepage.md | 2 +- .../Observability Provisioning.md | 2 +- .../Infrastructure/Observability.md | 2 +- .../Infrastructure/Prometheus.md | 2 +- .../Infrastructure/authentik-postgres.md | 24 - .../Infrastructure/authentik-server.md | 24 - .../Infrastructure/authentik-worker.md | 23 - .../LLM/Dankserver System Specs.md | 4 +- .../LLM/Dual-R9700-Cooperation.md | 4 +- .../LLM/FastFlowLM NPU Inference.md | 4 +- serverAppConfigs-docs/LLM/FreeToken Builds.md | 2 +- .../LLM/GGUF Compatibility.md | 2 +- .../LLM/Heterogeneous Split Triage.md | 2 +- .../LLM/LLM Bench Windows and Measurement.md | 2 +- .../LLM/LLM Performance Diagnosis.md | 2 +- .../LLM/Llama Cpp Multi-Device Splits.md | 2 +- .../LLM/Llama Cpp Serving Sizing.md | 2 +- .../LLM/Model Swap and Rollback.md | 2 +- .../LLM/PCIe Link and Atomics.md | 2 +- .../Ethernet Bandwidth Math.md | 118 +- .../Expected Performance.md | 6 +- .../Mixed-Vendor Experiments.md | 9 - .../Multi-Node Runtimes.md | 257 ++ .../Qwen3.6 27B Spec Methods.md | 9 - .../Qwen3.8 PP Deployment.md | 2 +- .../Pipeline Parallelism/Sample Script Set.md | 6 +- .../LLM/Pipeline Parallelism/index.md | 18 +- .../LLM/Pipeline Parallelism/llama.cpp RPC.md | 9 - .../vLLM Pipeline Parallelism.md | 9 - .../LLM/Quantization Production.md | 10 + .../LLM/RDNA4 Container Serving.md | 2 +- .../RDNA4 MoE Expert Offload VRAM Budget.md | 2 +- .../Running Qwen3.6 27B FP8 on Dankclaw.md | 2 +- ...nning Qwen3.6 35B A3B FP8 on Dankserver.md | 2 +- .../LLM/Tcclaviger Fork Ops.md | 2 +- serverAppConfigs-docs/LLM/VLLM Index.md | 2 +- .../LLM/dankclaw-iommu-translation-off.md | 4 +- .../dankserver-frontier-moe-feasibility.md | 4 +- .../LLM/dealign-ornith-crack.md | 5 +- serverAppConfigs-docs/LLM/deployment.md | 2 +- serverAppConfigs-docs/LLM/frank-archon.md | 10 +- serverAppConfigs-docs/LLM/frank-dragoon.md | 4 +- .../LLM/frank-medium-batch-ubatch.md | 6 +- .../LLM/frank-medium-llamacpp-dflash2.md | 6 +- .../LLM/frank-medium-model-swap.md | 13 +- .../LLM/frank-medium-radiance-mxfp4.md | 24 +- .../LLM/frank-medium-rocmfpx.md | 5 +- serverAppConfigs-docs/LLM/frank-medium.md | 65 + serverAppConfigs-docs/LLM/frank-npu.md | 3 +- .../LLM/frank-redo-2026-08-17.md | 4 +- serverAppConfigs-docs/LLM/frank-zealot.md | 3 +- .../LLM/freetoken-benchmark-sweep.md | 4 +- .../LLM/ikllama-ornith-dankserver.md | 7 +- serverAppConfigs-docs/LLM/index.md | 107 +- .../LLM/kingjones-q8agent-rocmfpx.md | 4 +- .../LLM/ling-3.0-flash-trial.md | 11 +- .../LLM/ling-3.0-flash-vllm-plan.md | 22 - .../LLM/qwen3.8-spec-matrix.md | 2 +- .../LLM/qwen3.8-strix-optimizations.md | 2 +- .../LLM/swift15-campaign-worklog.md | 10 + .../LLM/ubatch-benchmark-large.md | 4 +- ...Boot Forensics and Config Interrogation.md | 2 +- .../LLM/vLLM Porting and Compile Triage.md | 2 +- .../LLM/vllm_research_findings.md | 14 +- .../LLM/x4-pair-experiment.md | 2 +- .../Operations/Container Host Reachability.md | 2 +- .../Operations/Fleet Audit and Posture.md | 2 +- .../Operations/Fleet Ops Playbook.md | 2 +- .../Operations/Payload and Disk Reclaim.md | 2 +- .../Operations/Remote Job Runner.md | 2 +- .../Operations/SSH and Remote Access.md | 2 +- serverAppConfigs-docs/Operations/index.md | 25 +- .../AI Job Search and Resume Tools.md | 4 +- .../Research/ComfyUI ROCm and Vulkan.md | 2 +- .../Research/Llama Cpp Code Diff.md | 4 +- .../Research/Llama Cpp Runtimes.md | 4 +- .../Research/Local TTS and Voice Cloning.md | 4 +- .../Most Popular Self-Hosted AI Apps.md | 4 +- .../Research/PICO-8 Asset Standards.md | 2 +- serverAppConfigs-docs/Research/SGLang.md | 4 +- .../Research/Service Research Brief.md | 2 +- serverAppConfigs-docs/Research/index.md | 25 +- serverAppConfigs-docs/index.md | 228 +- serverAppConfigs-docs/log.md | 2513 +++++++++-------- 122 files changed, 2412 insertions(+), 2037 deletions(-) create mode 100644 serverAppConfigs-docs/Infrastructure/Authentik Outpost Reference.md create mode 100644 serverAppConfigs-docs/Infrastructure/Authentik Processes.md delete mode 100644 serverAppConfigs-docs/Infrastructure/authentik-postgres.md delete mode 100644 serverAppConfigs-docs/Infrastructure/authentik-server.md delete mode 100644 serverAppConfigs-docs/Infrastructure/authentik-worker.md delete mode 100644 serverAppConfigs-docs/LLM/Pipeline Parallelism/Mixed-Vendor Experiments.md create mode 100644 serverAppConfigs-docs/LLM/Pipeline Parallelism/Multi-Node Runtimes.md delete mode 100644 serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.6 27B Spec Methods.md delete mode 100644 serverAppConfigs-docs/LLM/Pipeline Parallelism/llama.cpp RPC.md delete mode 100644 serverAppConfigs-docs/LLM/Pipeline Parallelism/vLLM Pipeline Parallelism.md create mode 100644 serverAppConfigs-docs/LLM/frank-medium.md delete mode 100644 serverAppConfigs-docs/LLM/ling-3.0-flash-vllm-plan.md diff --git a/serverAppConfigs-docs/Agent/Agent Docs and Edits.md b/serverAppConfigs-docs/Agent/Agent Docs and Edits.md index 8e58062..0104539 100644 --- a/serverAppConfigs-docs/Agent/Agent Docs and Edits.md +++ b/serverAppConfigs-docs/Agent/Agent Docs and Edits.md @@ -1,7 +1,7 @@ --- type: reference title: Agent Docs and Edits -description: Two self-inflicted damage classes — structurally broken markdown after parallel agent edits, and botched hashline PUT/CUT repairs — with the audit checks and recovery order for each. +description: "Two self-inflicted doc damage classes from parallel agent edits, with the audit checks and recovery order for each." tags: [agents, docs, markdown, hashline, edit-recovery] timestamp: 2026-10-18T00:00:00Z --- diff --git a/serverAppConfigs-docs/Agent/Command Center Dotfiles.md b/serverAppConfigs-docs/Agent/Command Center Dotfiles.md index de95761..617c658 100644 --- a/serverAppConfigs-docs/Agent/Command Center Dotfiles.md +++ b/serverAppConfigs-docs/Agent/Command Center Dotfiles.md @@ -1,7 +1,7 @@ --- type: reference title: Command Center Dotfiles -description: The ~/.omp repo is itself the deployment — stow-equivalent packages, full-replica rsync transport, the pre-overwrite audit for non-blank hosts, deploy.sh mechanics, and the fish/sshpass transport gotchas. +description: "How the dotfiles repo deploys: stow-equivalent packages, full-replica rsync, the pre-overwrite audit, transport gotchas." tags: [dotfiles, omp, deploy, rsync, blankpad] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/Agent/OMP Harness.md b/serverAppConfigs-docs/Agent/OMP Harness.md index 48eafda..4e44a74 100644 --- a/serverAppConfigs-docs/Agent/OMP Harness.md +++ b/serverAppConfigs-docs/Agent/OMP Harness.md @@ -1,7 +1,7 @@ --- type: reference title: OMP Harness -description: The omp agent harness as configured on this fleet — compaction chain arithmetic, per-model sampler routing via compat.extraBody, and the static-preamble/skill-library audits that keep session context lean. +description: "The omp harness as configured here: compaction chain arithmetic, per-model sampler routing, and the context-lean audits." tags: [omp, harness, compaction, models-yml, context, samplers] timestamp: 2026-10-18T00:00:00Z --- diff --git a/serverAppConfigs-docs/Agent/index.md b/serverAppConfigs-docs/Agent/index.md index d042b3c..26c4697 100644 --- a/serverAppConfigs-docs/Agent/index.md +++ b/serverAppConfigs-docs/Agent/index.md @@ -1,23 +1,12 @@ ---- -type: index -description: Pages about the agent harness itself — omp config, vault editing discipline, and the dotfiles repo that deploys the agent's own home. -tags: [agent, index] -timestamp: 2026-10-05T00:00:00-07:00 ---- - # Agent Pages about the agent's own tooling and context — how omp is configured, how docs get edited safely, and how the agent's home is deployed. -- [[OMP Harness]] — compaction chain arithmetic, per-model samplers via - `compat.extraBody`, preamble and skill-library audits -- [[Agent Docs and Edits]] — doc-structure audits after parallel edits; - hashline edit recovery -- [[Command Center Dotfiles]] — the `~/.omp` repo-as-deployment, rsync - transport, override packages +- [[OMP Harness]] — The omp harness as configured here: compaction chain arithmetic, per-model sampler routing, and the context-lean audits. +- [[Agent Docs and Edits]] — Two self-inflicted doc damage classes from parallel agent edits, with the audit checks and recovery order for each. +- [[Command Center Dotfiles]] — How the dotfiles repo deploys: stow-equivalent packages, full-replica rsync, the pre-overwrite audit, transport gotchas. ## Related -- [[Testing Policy]] — the seams these pages' checks ride - [LLM](../LLM/index.md) — the fleet these agents operate diff --git a/serverAppConfigs-docs/Architecture/Container Network Namespaces.md b/serverAppConfigs-docs/Architecture/Container Network Namespaces.md index 69ef69d..e07af61 100644 --- a/serverAppConfigs-docs/Architecture/Container Network Namespaces.md +++ b/serverAppConfigs-docs/Architecture/Container Network Namespaces.md @@ -1,8 +1,7 @@ --- type: reference title: Container Network Namespaces -description: Which local services are addressable from which Podman network namespace on - Dankclaw, and why bridge-network containers cannot dial host-netns loopback listeners. +description: "How Podman builds container networks here: the network inventory, the netns sharing mechanism, and the isolation limits." tags: - networking - podman diff --git a/serverAppConfigs-docs/Architecture/Deploy State Triage.md b/serverAppConfigs-docs/Architecture/Deploy State Triage.md index 4bdc511..6ab442a 100644 --- a/serverAppConfigs-docs/Architecture/Deploy State Triage.md +++ b/serverAppConfigs-docs/Architecture/Deploy State Triage.md @@ -1,7 +1,7 @@ --- type: reference title: Deploy State Triage -description: Reading and reconciling the deploy-state markers — stale marker vs live container, pending-service lockout, interrupted/raced deploys — without restarting healthy heavy endpoints. +description: "Reading deploy-state markers against live containers and deciding, safely, whether to redeploy." tags: [deployment, deploy-state, systemd, podman, triage] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/Architecture/Deployment Flow.md b/serverAppConfigs-docs/Architecture/Deployment Flow.md index 2b56363..44195be 100644 --- a/serverAppConfigs-docs/Architecture/Deployment Flow.md +++ b/serverAppConfigs-docs/Architecture/Deployment Flow.md @@ -1,5 +1,5 @@ --- -type: guide +type: howto title: Deployment Flow description: Service-scoped Git deployment for the two homelab hosts. tags: diff --git a/serverAppConfigs-docs/Architecture/E2E Suite.md b/serverAppConfigs-docs/Architecture/E2E Suite.md index 179bd23..a03d3f0 100644 --- a/serverAppConfigs-docs/Architecture/E2E Suite.md +++ b/serverAppConfigs-docs/Architecture/E2E Suite.md @@ -1,6 +1,6 @@ --- type: reference -description: "`tests/e2e/` drives the **live fleet** through a real browser: fresh contexts" +description: "The Playwright suite that drives the live fleet through a real browser, its project set, and the FrankBook conditional." tags: - architecture - e2e diff --git a/serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md b/serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md index c5f1b24..eaa5769 100644 --- a/serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md +++ b/serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md @@ -1,7 +1,7 @@ --- type: reference title: Quadlet Lifecycle -description: The full Quadlet discipline on this fleet — ownership auditing, compose-to-Quadlet cutover, drift/crash-loop diagnosis, the podman 5.x v1 generator quirks, log retrieval, and compose bring-up with ground truth. +description: "The Quadlet discipline: ownership audits, compose cutover, drift and crash-loop triage, generator quirks, log retrieval." tags: [podman, quadlet, systemd, boot-persistence, compose, drift] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/Architecture/Stale Signal Audit.md b/serverAppConfigs-docs/Architecture/Stale Signal Audit.md index 37c16cb..6ef56e3 100644 --- a/serverAppConfigs-docs/Architecture/Stale Signal Audit.md +++ b/serverAppConfigs-docs/Architecture/Stale Signal Audit.md @@ -1,7 +1,7 @@ --- type: reference title: Stale Signal Audit -description: The audit battery for surfaces that rot without failing — unreachable probes, vacuous assertions, invalid checkers, gates that never run, and prose invalidated by a change. Four passes found real defects every time. +description: "The audit battery for surfaces that rot without failing: dead probes, vacuous assertions, gates that never run." tags: [testing, observability, audit, signals, gates] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/Dankclaw/ComfyUI Qwen-Image Workflows.md b/serverAppConfigs-docs/Dankclaw/ComfyUI Qwen-Image Workflows.md index 70ee470..97a8f8b 100644 --- a/serverAppConfigs-docs/Dankclaw/ComfyUI Qwen-Image Workflows.md +++ b/serverAppConfigs-docs/Dankclaw/ComfyUI Qwen-Image Workflows.md @@ -1,6 +1,6 @@ --- type: howto -description: "Qwen-Image 2.1 workflows on Dankclaw: two proven txt2img workflows (art + legible text), the API recipe, and which models/LoRAs to add next." +description: "Proven Qwen-Image text-to-image workflows on Dankclaw: the recipes, the API call shape, and which models to add next." tags: - dankclaw - comfyui diff --git a/serverAppConfigs-docs/Dankclaw/Frank TTS Engine.md b/serverAppConfigs-docs/Dankclaw/Frank TTS Engine.md index 55ae8f3..981ea19 100644 --- a/serverAppConfigs-docs/Dankclaw/Frank TTS Engine.md +++ b/serverAppConfigs-docs/Dankclaw/Frank TTS Engine.md @@ -1,6 +1,6 @@ --- type: reference -description: "frank-tts-engine is the arm A TTS engine on Dankclaw: qwentts.cpp's resident tts-server with the 1.7 B base talker, on a private podman bridge behind frank-tts — deployed 2026-10-18, and it publishes no port on purpose." +description: "The engine behind the fleet's speech endpoint: its runtime, private bridge, and the surface it publishes." tags: - dankclaw - tts diff --git a/serverAppConfigs-docs/Dankclaw/Frank TTS.md b/serverAppConfigs-docs/Dankclaw/Frank TTS.md index a3c3df0..b0344c0 100644 --- a/serverAppConfigs-docs/Dankclaw/Frank TTS.md +++ b/serverAppConfigs-docs/Dankclaw/Frank TTS.md @@ -1,11 +1,13 @@ --- type: reference -description: "frank-tts is the fleet's cloned-voice speech endpoint on Dankclaw :7772 — the one voice surface consumers address, in front of frank-tts-engine; deployed 2026-10-18 and wired as the harness speech role." +description: "The fleet's cloned-voice speech endpoint and the one voice surface consumers address; wired as the harness speech role." tags: + - endpoint - dankclaw - tts - voice - quadlet +timestamp: 2026-10-19 --- # Frank TTS @@ -202,3 +204,4 @@ harness bundle and measured the day of the flip: | First request of the day is ~0.6 s slower than the rest — or every name returns 400 | the warm pass did not run; it carries `-`, so it cannot fail the unit. `journalctl --user -u frank-tts-engine` for the warmer's lines, then `podman exec frank-tts-engine /app/warm-voices.sh` — the registry is memory-only (law 1 of [[Frank TTS Engine]]) | | The unit never appears after a deploy | the generator dropped it: `frank-tts-engine.network` is missing from `~/.config/containers/systemd/` | | A consumer rejects `narrator1` by listing 29 names (`af_heart … bm_fable`) and no request ever reaches this endpoint | that consumer is synthesizing **in the browser** with Kokoro.js, whose voice pack is fixed — OpenWebUI 0.11.4 ships it as the `browser-kokoro` TTS engine; see [[OpenWebUI]] | +--- diff --git a/serverAppConfigs-docs/Dankclaw/Gitea.md b/serverAppConfigs-docs/Dankclaw/Gitea.md index f8abd5d..dac5aad 100644 --- a/serverAppConfigs-docs/Dankclaw/Gitea.md +++ b/serverAppConfigs-docs/Dankclaw/Gitea.md @@ -1,5 +1,5 @@ --- -type: configuration +type: reference title: Gitea description: Gitea deployment on Dankclaw with its web UI reverse-proxied by the canonical Caddy service. tags: diff --git a/serverAppConfigs-docs/Dankclaw/Hermes-WebUI.md b/serverAppConfigs-docs/Dankclaw/Hermes-WebUI.md index d1b8159..f790365 100644 --- a/serverAppConfigs-docs/Dankclaw/Hermes-WebUI.md +++ b/serverAppConfigs-docs/Dankclaw/Hermes-WebUI.md @@ -1,7 +1,7 @@ --- -type: configuration +type: reference title: Hermes-WebUI (Hermex server) -description: The hermes-webui container on Dankclaw — the API server that the Hermex iPhone app talks to, reachable over an SSH tunnel, from the whole LAN (it binds all interfaces), and from https://hermes.franksdank.cloud (internal-only vhost). +description: "The iPhone companion app for Hermes on Dankclaw and the ways it is reached from the LAN." tags: - hermes - hermes-webui diff --git a/serverAppConfigs-docs/Dankclaw/Marreta.md b/serverAppConfigs-docs/Dankclaw/Marreta.md index 71ce484..308a056 100644 --- a/serverAppConfigs-docs/Dankclaw/Marreta.md +++ b/serverAppConfigs-docs/Dankclaw/Marreta.md @@ -1,7 +1,7 @@ --- type: reference title: Marreta on Dankclaw -description: The content-cleaning proxy (manualdousuario/marreta, Laravel/serversideup fpm-nginx) deployed as a rootless Quadlet pair that shares SearXNG's Gluetun netns for egress and sits behind the Anubis + Authentik guard chain at marreta.franksdank.cloud. +description: "The content-cleaning proxy SearXNG borrows egress from: a quadlet pair in the VPN netns behind the guard chain." tags: - dankclaw - marreta diff --git a/serverAppConfigs-docs/Dankclaw/Open Notebook.md b/serverAppConfigs-docs/Dankclaw/Open Notebook.md index 96cb5ca..04f4cd1 100644 --- a/serverAppConfigs-docs/Dankclaw/Open Notebook.md +++ b/serverAppConfigs-docs/Dankclaw/Open Notebook.md @@ -1,6 +1,6 @@ --- type: reference -description: "open-notebook is the fleet's podcast generator on Dankclaw: notes and sources in, a multi-speaker episode out — script written by frank-zealot, voices rendered by frank-tts, no cloud provider anywhere in the path." +description: "The fleet's podcast generator: notes in, multi-speaker episode out, script and voices both served in-house." tags: - dankclaw - open-notebook diff --git a/serverAppConfigs-docs/Dankclaw/Seerr.md b/serverAppConfigs-docs/Dankclaw/Seerr.md index 291890a..f347c25 100644 --- a/serverAppConfigs-docs/Dankclaw/Seerr.md +++ b/serverAppConfigs-docs/Dankclaw/Seerr.md @@ -1,6 +1,6 @@ --- type: reference -description: "Seerr is the unified request manager for the *arr stack (the maintained `seerr-team/seerr` successor to the archived Overseerr/Jellyseerr line)." +description: "The request manager for the *arr stack — the maintained successor to the archived Overseerr line." tags: - dankclaw - seerr @@ -40,8 +40,8 @@ the next outpost reload), and the missing app bounced post-login to authentik's own user UI instead of Seerr. Both backfilled live: the three standard flows copied from a known-good provider, plus an application (`seerr-guard`) linked to the provider. Verified by the hardened SSO sweep -(`tests/e2e/specs/13-sso.spec.ts`) — see [[Authentik]] for the two -provisioning rules this incident established. +(`tests/e2e/specs/13-sso.spec.ts`) — see [[Authentik Outpost Reference]] for the +two provisioning rules this incident established. ## *arr wiring diff --git a/serverAppConfigs-docs/Dankclaw/Swap Configuration.md b/serverAppConfigs-docs/Dankclaw/Swap Configuration.md index e057e24..eab3eae 100644 --- a/serverAppConfigs-docs/Dankclaw/Swap Configuration.md +++ b/serverAppConfigs-docs/Dankclaw/Swap Configuration.md @@ -1,7 +1,7 @@ --- -type: configuration +type: reference title: Dankclaw Swap Configuration -description: The host swapfile on Dankclaw — 64 GiB at /swap.img on ext4, fstab-driven via the generated swap.img.swap unit. Covers why it was raised from 8 GiB, the safe resize ordering that avoids an OOM on a host whose MemAvailable is inflated by pinned model weights, and the swappiness caveat. +description: "The host swapfile: why it is sized as it is, the resize ordering that avoids an OOM, and the swappiness caveat." tags: - dankclaw - host diff --git a/serverAppConfigs-docs/Dankclaw/Voice Packs.md b/serverAppConfigs-docs/Dankclaw/Voice Packs.md index f25efc8..aebeafa 100644 --- a/serverAppConfigs-docs/Dankclaw/Voice Packs.md +++ b/serverAppConfigs-docs/Dankclaw/Voice Packs.md @@ -1,6 +1,6 @@ --- type: reference -description: "How a frank-tts voice pack is built on Dankclaw: the three files every pack must carry, the 10-second source-material gate scripts/build-voice-pack.sh enforces, how to add a fourth voice, and what to do when a pinned engine bump changes the extractor's output sizes." +description: "How a voice pack is built: the files every pack must carry, the source-material gate, adding a voice, engine-bump drift." tags: - dankclaw - frank-tts diff --git a/serverAppConfigs-docs/Dankserver/Copyparty.md b/serverAppConfigs-docs/Dankserver/Copyparty.md index 951102a..87fa51c 100644 --- a/serverAppConfigs-docs/Dankserver/Copyparty.md +++ b/serverAppConfigs-docs/Dankserver/Copyparty.md @@ -143,13 +143,13 @@ Copyparty v1.20.9+ supports WOPI (Web Application Open Platform Interface) for e ## Security Advisories -| Version | CVE | Issue | -|---------|-----|-------| -| v1.20.19 | GHSA-phv8-wgjp-g4p9 | FTP upload outside volume | -| v1.20.17 | GHSA-x5pq-m9p8-f4vx | dirkey/filekey confusion | -| v1.20.9 | CVE-2026-27948 / GHSA-62cr-6wp5-q43h | XSS via file upload | +| Version | Advisory | CVE | Issue | +|---------|----------|-----|-------| +| v1.20.19 | GHSA-phv8-wgjp-g4p9 | CVE-2026-104102 | FTP upload outside the defined volumes (limited path traversal) | +| v1.20.17 | GHSA-x5pq-m9p8-f4vx | CVE-2026-70657 | dirkey/filekey confusion | +| v1.20.9 | GHSA-62cr-6wp5-q43h | CVE-2026-27948 | reflected XSS via the `?setck=` URL parameter (not the upload path — the separate upload/DOM-XSS advisory is GHSA-m2jw-cj8v-937r) | -Upgrade to v1.20.19+ to cover all known CVEs. The dirkey fix (v1.20.17) matters if using both `dirkeys` and `filekeys`. +Upgrade to v1.20.19+ to cover all known advisories. The dirkey fix (v1.20.17) matters if using both `dirkeys` and `filekeys`. Each row's patched version is the advisory's own `first_patched_version`. ## Gotchas @@ -174,3 +174,10 @@ Upgrade to v1.20.19+ to cover all known CVEs. The dirkey fix (v1.20.17) matters - [[Dankserver Stack]] - [[Caddy]] — TLS termination and authentication in front - [[ARR-Stack]] — Copyparty as a viewer alongside automated download management + +## Citations + +- GitHub Security Advisories, `9001/copyparty` — + https://github.com/9001/copyparty/security/advisories (accessed 2026-10-19); + each row's patched version read from `first_patched_version` in the + `https://api.github.com/advisories/` response. diff --git a/serverAppConfigs-docs/Dankserver/Dankserver Stack.md b/serverAppConfigs-docs/Dankserver/Dankserver Stack.md index e8d51fe..31f3972 100644 --- a/serverAppConfigs-docs/Dankserver/Dankserver Stack.md +++ b/serverAppConfigs-docs/Dankserver/Dankserver Stack.md @@ -1,6 +1,6 @@ --- type: reference -description: "Dankserver (192.168.0.235) runs the media and library stack." +description: "Service inventory and routing overview for the media and library host." tags: - dankserver timestamp: 2026-09-03 diff --git a/serverAppConfigs-docs/Dankserver/Jellyfin.md b/serverAppConfigs-docs/Dankserver/Jellyfin.md index 8dd05fd..1af373a 100644 --- a/serverAppConfigs-docs/Dankserver/Jellyfin.md +++ b/serverAppConfigs-docs/Dankserver/Jellyfin.md @@ -1,7 +1,7 @@ --- type: reference title: Jellyfin -description: Jellyfin media server on Dankserver — quadlet unit, entropy-volume data layout, ports, client access, and hardware-transcode status. +description: "Jellyfin on Dankserver: quadlet unit, entropy-volume data layout, ports, client access, hardware-transcode status." tags: [jellyfin, dankserver, media-server] timestamp: 2026-09-03T00:00:00Z --- diff --git a/serverAppConfigs-docs/Dankserver/RomM-Scraping.md b/serverAppConfigs-docs/Dankserver/RomM-Scraping.md index 2e98e8a..8a573df 100644 --- a/serverAppConfigs-docs/Dankserver/RomM-Scraping.md +++ b/serverAppConfigs-docs/Dankserver/RomM-Scraping.md @@ -1,6 +1,6 @@ --- type: reference -description: "This guide covers metadata providers, cover art, ROM naming conventions, emulator integration, and the scraping workflow for RomM on Dankserver." +description: "Metadata providers, cover art, ROM naming and the scraping workflow behind the RomM library." tags: - dankserver - romm diff --git a/serverAppConfigs-docs/Dankserver/mariadb.md b/serverAppConfigs-docs/Dankserver/mariadb.md index 0776c72..256908a 100644 --- a/serverAppConfigs-docs/Dankserver/mariadb.md +++ b/serverAppConfigs-docs/Dankserver/mariadb.md @@ -1,7 +1,7 @@ --- type: reference title: MariaDB (RomM Backend) -description: MariaDB 11 database backend for RomM on Dankserver — quadlet unit, network alias, env-file secret handling, and backup path. +description: "The MariaDB backend for RomM: quadlet unit, network alias, env-file secret handling, and the backup path." tags: [mariadb, romm, dankserver] timestamp: 2026-09-03T00:00:00Z --- diff --git a/serverAppConfigs-docs/Dankserver/redis.md b/serverAppConfigs-docs/Dankserver/redis.md index fcdc1ea..ecb36c3 100644 --- a/serverAppConfigs-docs/Dankserver/redis.md +++ b/serverAppConfigs-docs/Dankserver/redis.md @@ -1,7 +1,7 @@ --- type: reference title: Redis (RomM Cache / Queue) -description: Redis 7-alpine cache and task-queue backend for RomM on Dankserver — quadlet unit, network alias topology, and backup path. +description: "The Redis cache and task queue for RomM: quadlet unit, network alias topology, and the backup path." tags: [redis, romm, dankserver] timestamp: 2026-09-03T00:00:00Z --- diff --git a/serverAppConfigs-docs/Desktop/Noctalia and Niri.md b/serverAppConfigs-docs/Desktop/Noctalia and Niri.md index 3fafdd4..097c5c3 100644 --- a/serverAppConfigs-docs/Desktop/Noctalia and Niri.md +++ b/serverAppConfigs-docs/Desktop/Noctalia and Niri.md @@ -1,7 +1,7 @@ --- type: reference title: Noctalia and Niri -description: The noctalia v5 desktop shell on niri (workstation + blankpad) — config layers, remote IPC over ssh, custom-palette recolor, offline template rendering, and output rotation/pinning. +description: "The noctalia shell on niri across workstations: config layers, remote IPC, custom palettes, output rotation and pinning." tags: [desktop, noctalia, niri, blankpad, workstation, ghostty] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/FrankBook-Pro/ARR-Stack.md b/serverAppConfigs-docs/FrankBook-Pro/ARR-Stack.md index d4147bc..4ffaf42 100644 --- a/serverAppConfigs-docs/FrankBook-Pro/ARR-Stack.md +++ b/serverAppConfigs-docs/FrankBook-Pro/ARR-Stack.md @@ -1,7 +1,7 @@ --- type: reference title: "*arr Stack" -description: "The *arr automation stack (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) — FrankBook-Pro only, every container on the gluetun WireGuard netns (all egress VPN): topology, wiring, auth posture, and gotchas." +description: "The *arr automation stack on FrankBook-Pro, every container on the VPN netns: topology, wiring, and gotchas." tags: [arr-stack, prowlarr, sonarr, radarr, lidarr, readarr, qbittorrent, frankbook-pro, dankserver] timestamp: 2026-09-04T00:00:00Z --- diff --git a/serverAppConfigs-docs/FrankBook-Pro/index.md b/serverAppConfigs-docs/FrankBook-Pro/index.md index bcf0d6a..ee926d3 100644 --- a/serverAppConfigs-docs/FrankBook-Pro/index.md +++ b/serverAppConfigs-docs/FrankBook-Pro/index.md @@ -7,10 +7,10 @@ Gluetun WireGuard netns. The host overview lives at [[FrankBook-Pro]] ## *arr Stack -- [[ARR-Stack]] — the full *arr stack: indexers, managers, and the download client -- [[sonarr]] — TV show management -- [[radarr]] — movie management -- [[lidarr]] — music management -- [[readarr]] — ebook/audiobook management -- [[prowlarr]] — indexer management -- [[qbittorrent]] — torrent download client +- [[ARR-Stack]] — The *arr automation stack on FrankBook-Pro, every container on the VPN netns: topology, wiring, and gotchas. +- [[sonarr]] — Sonarr TV manager in the *arr stack on FrankBook-Pro — live quadlet deployment, ports, volumes, and wiring facts. +- [[radarr]] — Radarr movie manager in the *arr stack on FrankBook-Pro — live quadlet deployment, ports, volumes, and wiring facts. +- [[lidarr]] — Lidarr music manager in the *arr stack on FrankBook-Pro — live quadlet deployment, ports, volumes, and wiring facts. +- [[readarr]] — The ebook and audiobook manager in the *arr stack: retired upstream, pinned image, parked download client. +- [[prowlarr]] — Indexer management for the *arr stack: feed routes, the sync model, the API surface, and verified live gotchas. +- [[qbittorrent]] — The download client for the *arr stack: pinned image, shared WireGuard netns, WebUI credential recipe. diff --git a/serverAppConfigs-docs/FrankBook-Pro/prowlarr.md b/serverAppConfigs-docs/FrankBook-Pro/prowlarr.md index a79e79c..b204f8a 100644 --- a/serverAppConfigs-docs/FrankBook-Pro/prowlarr.md +++ b/serverAppConfigs-docs/FrankBook-Pro/prowlarr.md @@ -1,7 +1,7 @@ --- type: reference title: Prowlarr -description: Prowlarr indexer-management layer for the *arr stack on FrankBook-Pro — feed routes, sync model, API surface, and verified live gotchas. +description: "Indexer management for the *arr stack: feed routes, the sync model, the API surface, and verified live gotchas." tags: [prowlarr, arr-stack, frankbook-pro] timestamp: 2026-09-03T00:00:00Z --- diff --git a/serverAppConfigs-docs/FrankBook-Pro/qbittorrent.md b/serverAppConfigs-docs/FrankBook-Pro/qbittorrent.md index 7cbf6f2..65a8446 100644 --- a/serverAppConfigs-docs/FrankBook-Pro/qbittorrent.md +++ b/serverAppConfigs-docs/FrankBook-Pro/qbittorrent.md @@ -1,7 +1,7 @@ --- type: reference title: qBittorrent -description: qBittorrent download client for the *arr stack — pinned 4.6.7 on FrankBook-Pro, shares the Gluetun WireGuard netns, PBKDF2 WebUI credential recipe. +description: "The download client for the *arr stack: pinned image, shared WireGuard netns, WebUI credential recipe." tags: [qbittorrent, arr-stack, frankbook-pro, gluetun] timestamp: 2026-09-03T00:00:00Z --- diff --git a/serverAppConfigs-docs/FrankBook-Pro/readarr.md b/serverAppConfigs-docs/FrankBook-Pro/readarr.md index 56f5693..1737792 100644 --- a/serverAppConfigs-docs/FrankBook-Pro/readarr.md +++ b/serverAppConfigs-docs/FrankBook-Pro/readarr.md @@ -1,18 +1,19 @@ --- type: reference title: Readarr -description: Readarr ebook/audiobook manager in the *arr stack on FrankBook-Pro — retired upstream, pinned image, parked qB download client. +description: "The ebook and audiobook manager in the *arr stack: retired upstream, pinned image, parked download client." tags: [readarr, arr-stack, frankbook-pro] timestamp: 2026-09-03T00:00:00Z --- # Readarr -> **RETIRED UPSTREAM (May 2026).** The Servarr Team retired Readarr: its -> metadata became unusable and the community migration to Open Library stalled. -> There is no upstream development. Alternatives: the Bookshelf community fork, -> or the Calibre content server. This instance still runs on a pinned image -> tag; see below. +> **RETIRED UPSTREAM (announced 2025-06-27).** The Servarr Team retired +> Readarr: its metadata became unusable and the community migration to Open +> Library stalled. There is no upstream development — the GitHub repository is +> archived and its last push is 2025-06-27. Alternatives: the Bookshelf +> community fork, or the Calibre content server. This instance still runs on a +> pinned image tag; see below. Readarr manages ebooks and audiobooks in the *arr stack. The stack runs on **FrankBook-Pro** (the Mac mini) as quadlet units inside its podman machine — @@ -69,9 +70,9 @@ Restart=always ## Gotchas -- Readarr is abandoned upstream (retired May 2026; last line was 0.4.x before - the final tags). The pinned `amd64-1.0.980` tag exists precisely because - `:latest` does not exist for this image on LSIO. +- Readarr is abandoned upstream (retired 2025-06-27; last line was 0.4.x + before the final tags). The pinned `amd64-1.0.980` tag exists precisely + because `:latest` does not exist for this image on LSIO. - Calibre integration is folder-based, not API-based. Readarr drops files into the books root which [[Calibre-Web]] serves. - Not all indexers carry all formats (epub, mobi, pdf). Goodreads/StoryGraph @@ -81,3 +82,10 @@ Restart=always - [[ARR-Stack]] — full *arr stack documentation - [[Calibre-Web]] — ebook web frontend that serves Readarr downloads + +## Citations + +- Servarr wiki, *Readarr* (Retired) — "Announcement: Retirement of Readarr + 2025-06-27" — https://wiki.servarr.com/readarr (accessed 2026-10-19) +- `Readarr/Readarr` on GitHub: `archived: true`, last push 2025-06-27 — + https://api.github.com/repos/Readarr/Readarr (accessed 2026-10-19) diff --git a/serverAppConfigs-docs/Infrastructure/Alertmanager.md b/serverAppConfigs-docs/Infrastructure/Alertmanager.md index f33aefd..2984935 100644 --- a/serverAppConfigs-docs/Infrastructure/Alertmanager.md +++ b/serverAppConfigs-docs/Infrastructure/Alertmanager.md @@ -1,6 +1,6 @@ --- type: reference -description: "Alertmanager receives firing rules from Prometheus, groups and dedupes them, and forwards each group as a webhook to the host-side Signal relay — which sends it through..." +description: "Receives firing rules from Prometheus, groups and dedupes them, and forwards each group to the host-side Signal relay." tags: - infrastructure timestamp: 2026-09-17 diff --git a/serverAppConfigs-docs/Infrastructure/Authentik Outpost Reference.md b/serverAppConfigs-docs/Infrastructure/Authentik Outpost Reference.md new file mode 100644 index 0000000..6139884 --- /dev/null +++ b/serverAppConfigs-docs/Infrastructure/Authentik Outpost Reference.md @@ -0,0 +1,230 @@ +--- +type: reference +description: "Outpost, proxy-provider, OAuth2, LDAP and flow configuration for the homelab's Authentik SSO." +tags: + - infrastructure + - authentik +timestamp: 2026-10-19 +--- + +# Authentik Outpost Reference + +Configuration lookup for the [[Authentik]] deployment: the live proxy-provider +inventory behind the guarded vhosts, the provisioning rules the embedded outpost +enforces, this fork's OAuth2 relying-party quirks, the LDAP provider objects, the +self-serve user-settings flow, and the live database shape. What the service is, +how it is deployed, wired and operated is [[Authentik]]; the three core +containers are [[Authentik Processes]]. + +## Live proxy-provider inventory + +Live provider inventory (read from the `authentik-postgres` container with +read-only `SELECT`s, 2026-10-18). Nine proxy providers, all attached to outpost +`7e39655b-58d3-4fe6-a80a-513f31051217` ("authentik Embedded Outpost", 9 +members in `authentik_outposts_outpost_providers`), all with +`internal_host = http://caddy:8017`, all carrying the **same three flow +UUIDs** (`default-authentication-flow`, +`default-provider-authorization-implicit-consent`, +`default-provider-invalidation-flow`), and each backed by an Application +(the `slug` column below): + +| Provider pk | name | external_host | app slug | +|---|---|---|---| +| 1 | embedded-outpost | `https://www.franksdank.cloud` | caddy | +| 2 | search-proxy | `https://search.franksdank.cloud` | search | +| 4 | prometheus-proxy | `https://prometheus.franksdank.cloud` | prometheus | +| 5 | romm-proxy | `https://romm.franksdank.cloud` | romm | +| 6 | files-proxy | `https://files.franksdank.cloud` | files | +| 7 | loki-proxy | `https://loki.franksdank.cloud/` | loki | +| 8 | grafana-proxy | `https://grafana.franksdank.cloud/` | grafana-guard | +| 10 | seerr-proxy | `https://seerr.franksdank.cloud/` | seerr-guard | +| 13 | marreta-proxy | `https://marreta.franksdank.cloud` | marreta | + +That is exactly the Caddyfile's guarded set (`dankclaw/Caddyfile:283-423`), so +the lockstep rule in [[Authentik]] holds today. Two corrections against the 2026-09-02 table: +**pk 13 (`marreta-proxy`) was added** when Marreta went onto the chain, and +**pk 3 (`hermes-proxy`) no longer exists at all** — `select … where id=3` +returns zero rows, so it is deleted, not "dormant"; its vhost survived but is +exempt-by-construction (`dankclaw/Caddyfile:179-191`). OAuth2 relying parties, +not proxy providers, so the proxy outpost does not deserialize them: pk 9 +`grafana` (`grant_types = {authorization_code}`), pk 12 `Seerr OIDC` +(`grant_types = {}` — empty, i.e. still in the state the token-endpoint rule +below rejects), and pk 14 `Open WebUI` +(`grant_types = {authorization_code,refresh_token}`). Trailing slashes on +`external_host` (the runbook-recipe style) and bare origins both intercept +correctly — the outpost matches on hostname. + +## Provisioning rules for proxy providers + +Two provisioning rules, each verified live by an outage: + +1. **All three flows are required.** A ProxyProvider created via raw API/ORM + with any of `authentication_flow`/`authorization_flow`/`invalidation_flow` + null wedges the embedded outpost's config fetch: the Rust side rejects the + whole config JSON (serde: `invalid type: null, expected a formatted UUID + string`), logs `Failed to fetch outpost from API, retrying in 3 seconds` + forever, and **everything behind the outpost dies at the next outpost + reload** — including hosts whose providers were fine (verified 2026-09-02 + when three providers provisioned without any flows took down every guarded + vhost after a server restart). Backfill all three flows (copy from pk 1: + `default-authentication-flow`, + `default-provider-authorization-implicit-consent` (the implicit-consent + authorization flow, `default-invalidation-flow`) and the retry loop + recovers on its next 3-second cycle. The silent killer: while the outpost + process keeps its last good in-memory config, everything works — the + outage only surfaces on restart/reload. +2. **Every proxy provider needs an attached Application.** The outpost + resolves the post-login redirect through the provider's application; a + provider with no app completes the login (authentik session established) + but bounces to authentik's own user UI (`/if/user/#/library`) instead of + the app. No error is logged anywhere — the SSO "works" and the app never + appears. Create the Application with `provider_id` set (the UI does this + automatically; raw API/ORM drives must do it explicitly). + +## OAuth2 relying-party quirks (this fork, verified live 2026-08-31) + +The Grafana RP provider (`grafana`, pk 9, confidential client) was created and +verified end-to-end via `ak shell` ORM drives. This fork's OAuth2 provider +implementation diverges from upstream in ways that bit us; all verified live: + +- **Application row required.** `OAuth2Provider.objects.create()` does not + create the single-table-inheritance `Application` row; without it the flow + executor can never complete an authorize request. Create both rows. +- **`grant_types` defaults to empty.** Until the provider's `grant_types` + contains `"authorization_code"` (enum value with underscore), the token + endpoint rejects every exchange with `invalid_request`. +- **Token endpoint is form-encoded + Basic-auth only.** A JSON body makes + `parse_token_request` see no `grant_type` → `unsupported_grant_type`; + POST-body client secrets are not accepted — authenticate the confidential + client with `Authorization: Basic base64(client_id:client_secret)`. +- **Provider resolution is by `client_id` query param** on the global + `/application/o/authorize/` route; per-slug authorize paths do not exist in + this build. Token/userinfo live at `/application/o/token/` and + `/application/o/userinfo/?api=1`. +- **Property-mapping evaluator is broken → userinfo returns `sub` only.** + `PropertyMappingEvaluator.evaluate(*args, **kwargs)` forwards kwargs to + `BaseEvaluator.evaluate(expression_source)`, so the first context kwarg + becomes the *expression source* instead of the context; every call raises a + swallowed `PropertyMappingExpressionException` and the claim is skipped. + Scope-grant filtering has the same class of bug: `ScopeMapping` has no + provider FK (linkage lives in the `ProviderPropertyMapping` through model), + so `filter(provider=...)` always yields an empty queryset and tokens asked + to carry scopes get zero. Net effect: userinfo/introspection answers with + `{sub}` only — email/groups/entitlements claims cannot be delivered by this + build until the evaluator is patched or upgraded. Clients consuming this + provider must key identity off `sub`. +- Scope mappings for this provider exist — `grafana-email-scope` and + `grafana-profile-scope` are both rows in `authentik_core_propertymapping` + (read-only `SELECT`, 2026-10-18) — and will start emitting claims the moment + the evaluator bug is fixed; no config change is needed then. + +## LDAP provider objects (provisioned via `ak shell` ORM) + +The LDAP outpost's own deployment — unit, image digest, published ports, token +env var — is in [[Authentik]]. The objects it serves: + +Read from the live database (read-only `SELECT`s against the +`authentik-postgres` container, 2026-10-18): + +| Object | Value | +|---|---| +| LDAP provider | `Jellyfin LDAP` (pk 11), base DN `dc=franksdank,dc=cloud`, `search_mode=direct`, `bind_mode=direct` | +| Bind DN | `cn=svc-jellyfin-ldap,ou=users,dc=franksdank,dc=cloud` *(unverified 2026-10-18: base DN confirmed in the DB, the `ou=users` path was not probed — an LDAP bind needs the bind secret)* | +| Service account | `svc-jellyfin-ldap` (user pk 8). **It is not passwordless**: its password field is a real hash, not Django's `!`-prefixed unusable marker (`starts_with(password,'!')` → `f`) — an earlier revision's "passwordless/unusable" claim is wrong, which matters because an account with a usable password can sign in through the ordinary password flow *(unverified 2026-10-18: whether anyone holds that password)* | +| Bind token | exactly one row for pk 8, **`intent=app_password`**, non-expiring — an earlier revision recorded `intent=verification`; the live value is `app_password` | +| RBAC role | `svc-jellyfin-ldap-search`, carrying exactly one object-level permission `authentik_providers_ldap.search_full_directory` (verified via `guardian_roleobjectpermission`) | +| Groups | `jellyfin_users`, `jellyfin_admins` (both exist in `authentik_core_group`) | +| Application | name `Jellyfin`, **slug `jellyfin-ldap`** (an earlier revision said slug `jellyfin`, which does not exist) | + +Hard-won rules (the first is now contested by the live database): + +- **Token intent: contested, do not trust either value blindly.** An earlier + revision asserted that a bind token with `intent=app_password` is rejected by + the flow's password stage with `Invalid password` and only + `intent=verification` passes. The live token for `svc-jellyfin-ldap` is + `app_password` (`authentik_core_token`, 2026-10-18), so either the rule is + wrong or the current Jellyfin bind path is not that token. Nothing observable + settles it today: the LDAP outpost's retained log holds only its 5-minute + `initialized direct binder` / `Update providers` config cycles and no bind + activity *(unverified 2026-10-18: no bind was performed — reproducing it + requires the bind secret)*. Re-verify with a real bind before changing the + token's intent either way. +- **Search needs `search_full_directory`.** Without the RBAC grant the + outpost returns only the bind user itself — a `(memberOf=…)` filter then + matches zero entries and the plugin logs `Found no users matching X`. + The permission is object-level on the provider; grant it via an RBAC role + and retest. +- `ak shell` import path for the provider model in 2026.8: + `from authentik.providers.ldap.models import LDAPProvider`. + *(unverified 2026-10-18: importing it outside a configured Django shell fails + with `ImproperlyConfigured`, and no `ak shell` was driven against the live + auth service for this check)* + +## Self-serve user-settings flow (password change) + +Users change their own account details through the `default-user-settings-flow` +(`https://auth.franksdank.cloud/if/flow/default-user-settings-flow/`) — the +target of the homepage footer's "account" link. It is a `stage_configuration` +designated flow, so it renders every bound `ConfigurableStage` for the logged-in +user. + +Stages, in execution order (the `FlowStageBinding.order` field, not the M2M +insertion order): + +| order | stage | type | what it does | +|---|---|---|---| +| 20 | `default-user-settings` | `PromptStage` | name / email / locale **+ password** | +| 100 | `default-user-settings-write` | `UserWriteStage` | writes the prompt data to the user | + +**Password change is a prompt field, not a stage.** The `UserWriteStage` +writes whatever the preceding `PromptStage` collected: for the key `password` +it calls `user.set_password(value)` (`stages/user_write/stage.py`). So the fix +is a `password` field on the `default-user-settings` prompt +(`field_key=password`, `type=password`, `required=False`, order 204) — NOT a +`PasswordStage`. A `PasswordStage` is an *authentication* stage: its +`validate_password` runs `authenticate()` against the backends to check the +*current* password; it never sets a new one. (A `PasswordStage` was added on +2026-09-25 and removed the same day for exactly this reason.) + +Two ORM gotchas: `Flow.stages.add()` fails on this through model +(`Can't bulk create a multi-table inherited model` — `FlowStageBinding` is +MTI), so create the `FlowStageBinding` row directly with an explicit `order`; +and a prompt field is a `Prompt` row linked through `PromptStageField` +(`prompt_stage` + `prompt`; the order comes from `Prompt.order`). + +**A blank password must not set an empty password.** `set_password("")` would +make the account log in with no password, so a validation policy +(`default-user-settings-no-empty-password`, bound to the prompt via +`PromptStageValidationPolicy`) denies the form when `prompt_data["password"]` +is `""` and allows it when set. Verified with the `PolicyEvaluator` +(empty → deny, set → allow, absent → allow) and end-to-end by driving +`UserWriteStageView.update_user` on a throwaway user (new password valid, old +invalid). + +## Live database shape + +For live DB truth prefer the postgres container, and it works without any +credential — the local socket is trusted, so a read-only audit is just +`podman exec -i authentik-postgres psql -U authentik -d authentik -At -f -` +with SQL on stdin (quoting SQL through ssh+zsh otherwise fights you). Verify +before guessing: 2026.x renamed both tables and columns, and the models are +MTI all the way down — `authentik_core_provider` holds `id`, `name` and the +three `*_flow_id` columns; the proxy table is +`authentik_providers_proxy_proxyprovider` (doubled `proxy`) and its PK column +really is **`oauth2provider_ptr_id`** (proxy providers inherit the OAuth2 +table in 2026.8), the LDAP table is +`authentik_providers_ldap_ldapprovider` with PK `provider_ptr_id`; +`authentik_core_application` attaches to a provider through `provider_id` +(its PK is `policybindingmodel_ptr_id`, not `id`), and `authentik_outposts_outpost` +is keyed by `uuid`. So `Provider.objects.filter(type=…)` raising `FieldError` +is the same story from the ORM side: `type` lives in the subclass tables. +Never select secret columns (`client_secret`, `key`, `password`) into a log. + +## Cross-references + +- [[Authentik]] — what the service is, its deployment, wiring, operations and + failure modes +- [[Authentik Processes]] — the postgres, server and worker containers +- [[Caddy]] — the reverse proxy carrying the `:8016`/`:8017` auth-chain listeners +- [[Jellyfin]] — consumes the LDAP outpost via its LDAP-Auth plugin +- [[Grafana]] — guarded vhost *and* OAuth2 relying party of this provider diff --git a/serverAppConfigs-docs/Infrastructure/Authentik Processes.md b/serverAppConfigs-docs/Infrastructure/Authentik Processes.md new file mode 100644 index 0000000..a911a02 --- /dev/null +++ b/serverAppConfigs-docs/Infrastructure/Authentik Processes.md @@ -0,0 +1,76 @@ +--- +type: reference +description: "The three core Authentik containers — database, server and worker — with their units, images, roles and probes." +tags: + - infrastructure + - authentik + - postgres +timestamp: 2026-10-19 +--- + +# Authentik Processes + +The three core containers of the [[Authentik]] stack on Dankclaw. The two +outpost units are not covered here — they live on [[Authentik]] and +[[Authentik Outpost Reference]]. Server and worker run the same image; the +`command` selects the role (`server` / `worker`). Data persists under +`/home/dank/Containers/authentik/` as bind mounts (repo policy — no named +volumes), moved from `/home/dank/authentik/` in the 2026-09-01 quadlet +standardization; the retired stubs this page replaces recorded the pre-move +paths `/home/dank/authentik/data` (server and worker) and +`/home/dank/authentik/postgres` (database). + +## authentik-postgres + +The PostgreSQL 16 database for Authentik — the only stateful store in the stack +(users, flows, sessions, events). Sessions, caching, and WebSockets moved to +PostgreSQL in 2025.10 (Redis removed). + +- **Image:** `docker.io/library/postgres:16-alpine` +- **Data:** `/home/dank/Containers/authentik/postgres` (bind mount; no named + volumes) +- **Health:** `pg_isready -d authentik -U authentik` +- **Retired wording:** the stub called this a "Compose service … in + `dankclaw/compose.yaml`"; the compose stack is gone and the unit is a quadlet. +- The side-by-side with `gitea-postgres` (network, published ports, secret + source, boot persistence) is owned by [[Postgres]]. + +## authentik-server + +The Authentik server — the core, the admin API, and the embedded Caddy outpost. +It also serves the login page at `https://auth.franksdank.cloud`. + +- **Unit:** `dankclaw/authentik/quadlet/authentik-server.container` + (`authentik-server.service`) +- **Image:** `ghcr.io/goauthentik/server:2026.8.3` (`command: server`) +- **Port:** `9000` (published) — the outpost, the admin API and the login UI all + live here +- **Data:** `/home/dank/Containers/authentik/data` (bind mount, shared with the + worker) +- **Health:** recorded by the retired stub as + `http://localhost:9000/-/health/`; superseded — [[Authentik]] → Operations + owns the live probe. +- **Retired wording:** the stub described "the Caddy `forward_auth` vhosts" + proxying here; the chain Caddy actually runs is [[Authentik]] → Caddy + integration. + +## authentik-worker + +The Authentik worker — background tasks: event processing, provider +synchronization, scheduled flows, and certificate renewal. + +- **Unit:** `dankclaw/authentik/quadlet/authentik-worker.container` + (`authentik-worker.service`) +- **Image:** `ghcr.io/goauthentik/server:2026.8.3` (`command: worker`) +- **Runs as:** `0:0` (rootless: container root maps to the host user that owns + the data dir) +- **Data:** `/home/dank/Containers/authentik/data` (bind mount, shared with the + server) +- **No port** — the worker has no HTTP endpoint. + +## Cross-references + +- [[Authentik]] — what the service is, its deployment, wiring, operations and + failure modes +- [[Authentik Outpost Reference]] — provider, flow and token configuration +- [[Postgres]] — both PostgreSQL clusters on Dankclaw, side by side diff --git a/serverAppConfigs-docs/Infrastructure/Authentik.md b/serverAppConfigs-docs/Infrastructure/Authentik.md index 78d10e5..2d1578f 100644 --- a/serverAppConfigs-docs/Infrastructure/Authentik.md +++ b/serverAppConfigs-docs/Infrastructure/Authentik.md @@ -16,7 +16,7 @@ OpenWebUI, Jellyfin (LDAP outpost), the *arr stack, Pi-hole (basicauth edge) — they handle their own session. Grafana is the exception that is both: it is a guarded vhost (`grafana-proxy`, pk 8, locked to Authentik SSO) *and* an OAuth2 relying party (`grafana`, pk 9) (`dankclaw/Caddyfile:122-132`, -`dankclaw/Caddyfile:374-377`; live provider table below). +`dankclaw/Caddyfile:374-377`; live provider table in [[Authentik Outpost Reference]]). - **Version:** `2026.8.3` (pinned in `dankclaw/authentik/quadlet/*.container`; the outpost's boot log reads `"Booting authentik" … "version": "2026.8.3"`, @@ -35,6 +35,13 @@ relying party (`grafana`, pk 9) (`dankclaw/Caddyfile:122-132`, pre-quadlet era, retired. - **LDAP outpost:** serves LDAP for Jellyfin — see the section below. +**Split off from this page:** [[Authentik Processes]] owns the per-container +facts for `authentik-postgres`, `authentik-server` and `authentik-worker`; +[[Authentik Outpost Reference]] owns the provider/flow/token lookup material — +the live proxy-provider inventory, the proxy-provider provisioning rules, the +OAuth2 relying-party quirks, the LDAP provider objects, the user-settings flow +design, and the live database shape. + ## Why Authentik The comparison that chose it (against Authelia, Keycloak, Zitadel, Ory) was @@ -100,23 +107,10 @@ contract: The chain terminates there — nothing dials back into Authentik from `:8017`. -## The three compose services +## The stack's processes -| Service | Role | Image | -|---|---|---| -| `authentik-postgres` | PostgreSQL 16 (users, flows, sessions, events) | `docker.io/library/postgres:16-alpine` | -| `authentik-server` | Core + admin API + the embedded Caddy outpost (port 9000) | `ghcr.io/goauthentik/server:2026.8.3` | -| `authentik-worker` | Background tasks (event processing, provider sync, flows) | `ghcr.io/goauthentik/server:2026.8.3` | - -- **Postgres** is the only stateful store. Sessions, caching, and WebSockets - moved to PostgreSQL in 2025.10 (Redis removed). -- **Server** and **worker** run the same image; the `command` selects the role - (`server` / `worker`). -- The worker runs as `0:0` (rootless: container root maps to the host user that - owns the data dir). -- Data persists under `/home/dank/Containers/authentik/{data,postgres}` - (bind mounts, repo policy — no named volumes; moved from `/home/dank/authentik/` - in the 2026-09-01 quadlet standardization). +Per-container facts — unit path, image and role, port, run-as, data directory, +health probe — are owned by [[Authentik Processes]]. ## Caddy integration @@ -142,104 +136,9 @@ routing table, and the Authentik proxy-provider list must be the same set of hosts. A guarded host with no provider 404s at login time; a provider with no DNS record breaks off-LAN clients. -Live provider inventory (read from the `authentik-postgres` container with -read-only `SELECT`s, 2026-10-18). Nine proxy providers, all attached to outpost -`7e39655b-58d3-4fe6-a80a-513f31051217` ("authentik Embedded Outpost", 9 -members in `authentik_outposts_outpost_providers`), all with -`internal_host = http://caddy:8017`, all carrying the **same three flow -UUIDs** (`default-authentication-flow`, -`default-provider-authorization-implicit-consent`, -`default-provider-invalidation-flow`), and each backed by an Application -(the `slug` column below): - -| Provider pk | name | external_host | app slug | -|---|---|---|---| -| 1 | embedded-outpost | `https://www.franksdank.cloud` | caddy | -| 2 | search-proxy | `https://search.franksdank.cloud` | search | -| 4 | prometheus-proxy | `https://prometheus.franksdank.cloud` | prometheus | -| 5 | romm-proxy | `https://romm.franksdank.cloud` | romm | -| 6 | files-proxy | `https://files.franksdank.cloud` | files | -| 7 | loki-proxy | `https://loki.franksdank.cloud/` | loki | -| 8 | grafana-proxy | `https://grafana.franksdank.cloud/` | grafana-guard | -| 10 | seerr-proxy | `https://seerr.franksdank.cloud/` | seerr-guard | -| 13 | marreta-proxy | `https://marreta.franksdank.cloud` | marreta | - -That is exactly the Caddyfile's guarded set (`dankclaw/Caddyfile:283-423`), so -the lockstep rule holds today. Two corrections against the 2026-09-02 table: -**pk 13 (`marreta-proxy`) was added** when Marreta went onto the chain, and -**pk 3 (`hermes-proxy`) no longer exists at all** — `select … where id=3` -returns zero rows, so it is deleted, not "dormant"; its vhost survived but is -exempt-by-construction (`dankclaw/Caddyfile:179-191`). OAuth2 relying parties, -not proxy providers, so the proxy outpost does not deserialize them: pk 9 -`grafana` (`grant_types = {authorization_code}`), pk 12 `Seerr OIDC` -(`grant_types = {}` — empty, i.e. still in the state the token-endpoint rule -below rejects), and pk 14 `Open WebUI` -(`grant_types = {authorization_code,refresh_token}`). Trailing slashes on -`external_host` (the runbook-recipe style) and bare origins both intercept -correctly — the outpost matches on hostname. - -Two provisioning rules, each verified live by an outage: - -1. **All three flows are required.** A ProxyProvider created via raw API/ORM - with any of `authentication_flow`/`authorization_flow`/`invalidation_flow` - null wedges the embedded outpost's config fetch: the Rust side rejects the - whole config JSON (serde: `invalid type: null, expected a formatted UUID - string`), logs `Failed to fetch outpost from API, retrying in 3 seconds` - forever, and **everything behind the outpost dies at the next outpost - reload** — including hosts whose providers were fine (verified 2026-09-02 - when three providers provisioned without any flows took down every guarded - vhost after a server restart). Backfill all three flows (copy from pk 1: - `default-authentication-flow`, - `default-provider-authorization-implicit-consent` (the implicit-consent - authorization flow, `default-invalidation-flow`) and the retry loop - recovers on its next 3-second cycle. The silent killer: while the outpost - process keeps its last good in-memory config, everything works — the - outage only surfaces on restart/reload. -2. **Every proxy provider needs an attached Application.** The outpost - resolves the post-login redirect through the provider's application; a - provider with no app completes the login (authentik session established) - but bounces to authentik's own user UI (`/if/user/#/library`) instead of - the app. No error is logged anywhere — the SSO "works" and the app never - appears. Create the Application with `provider_id` set (the UI does this - automatically; raw API/ORM drives must do it explicitly). - - -## OAuth2 relying-party quirks (this fork, verified live 2026-08-31) - -The Grafana RP provider (`grafana`, pk 9, confidential client) was created and -verified end-to-end via `ak shell` ORM drives. This fork's OAuth2 provider -implementation diverges from upstream in ways that bit us; all verified live: - -- **Application row required.** `OAuth2Provider.objects.create()` does not - create the single-table-inheritance `Application` row; without it the flow - executor can never complete an authorize request. Create both rows. -- **`grant_types` defaults to empty.** Until the provider's `grant_types` - contains `"authorization_code"` (enum value with underscore), the token - endpoint rejects every exchange with `invalid_request`. -- **Token endpoint is form-encoded + Basic-auth only.** A JSON body makes - `parse_token_request` see no `grant_type` → `unsupported_grant_type`; - POST-body client secrets are not accepted — authenticate the confidential - client with `Authorization: Basic base64(client_id:client_secret)`. -- **Provider resolution is by `client_id` query param** on the global - `/application/o/authorize/` route; per-slug authorize paths do not exist in - this build. Token/userinfo live at `/application/o/token/` and - `/application/o/userinfo/?api=1`. -- **Property-mapping evaluator is broken → userinfo returns `sub` only.** - `PropertyMappingEvaluator.evaluate(*args, **kwargs)` forwards kwargs to - `BaseEvaluator.evaluate(expression_source)`, so the first context kwarg - becomes the *expression source* instead of the context; every call raises a - swallowed `PropertyMappingExpressionException` and the claim is skipped. - Scope-grant filtering has the same class of bug: `ScopeMapping` has no - provider FK (linkage lives in the `ProviderPropertyMapping` through model), - so `filter(provider=...)` always yields an empty queryset and tokens asked - to carry scopes get zero. Net effect: userinfo/introspection answers with - `{sub}` only — email/groups/entitlements claims cannot be delivered by this - build until the evaluator is patched or upgraded. Clients consuming this - provider must key identity off `sub`. -- Scope mappings for this provider exist — `grafana-email-scope` and - `grafana-profile-scope` are both rows in `authentik_core_propertymapping` - (read-only `SELECT`, 2026-10-18) — and will start emitting claims the moment - the evaluator bug is fixed; no config change is needed then. +The live proxy-provider inventory (the per-vhost `external_host` → app mapping), +the two provisioning rules the outpost enforces, and this fork's OAuth2 +relying-party quirks are reference lookup: [[Authentik Outpost Reference]]. ## LDAP outpost (Jellyfin auth, live 2026-09-01) @@ -267,44 +166,9 @@ rule at the current `2026.8.3` pin. Resolve once with `podman manifest inspect (2026-10-18). The outpost token travels in `AUTHENTIK_LDAP_OUTPOST_TOKEN` from the host `dankclaw/.env` (git-ignored). -### Authentik-side objects (provisioned via `ak shell` ORM) - -Read from the live database (read-only `SELECT`s against the -`authentik-postgres` container, 2026-10-18): - -| Object | Value | -|---|---| -| LDAP provider | `Jellyfin LDAP` (pk 11), base DN `dc=franksdank,dc=cloud`, `search_mode=direct`, `bind_mode=direct` | -| Bind DN | `cn=svc-jellyfin-ldap,ou=users,dc=franksdank,dc=cloud` *(unverified 2026-10-18: base DN confirmed in the DB, the `ou=users` path was not probed — an LDAP bind needs the bind secret)* | -| Service account | `svc-jellyfin-ldap` (user pk 8). **It is not passwordless**: its password field is a real hash, not Django's `!`-prefixed unusable marker (`starts_with(password,'!')` → `f`) — the page's earlier "passwordless/unusable" claim is wrong, which matters because an account with a usable password can sign in through the ordinary password flow *(unverified 2026-10-18: whether anyone holds that password)* | -| Bind token | exactly one row for pk 8, **`intent=app_password`**, non-expiring — the page previously recorded `intent=verification`; the live value is `app_password` | -| RBAC role | `svc-jellyfin-ldap-search`, carrying exactly one object-level permission `authentik_providers_ldap.search_full_directory` (verified via `guardian_roleobjectpermission`) | -| Groups | `jellyfin_users`, `jellyfin_admins` (both exist in `authentik_core_group`) | -| Application | name `Jellyfin`, **slug `jellyfin-ldap`** (the page previously said slug `jellyfin`, which does not exist) | - -Hard-won rules (the first is now contested by the live database): - -- **Token intent: contested, do not trust either value blindly.** This page - long asserted that a bind token with `intent=app_password` is rejected by the - flow's password stage with `Invalid password` and only `intent=verification` - passes. The live token for `svc-jellyfin-ldap` is `app_password` - (`authentik_core_token`, 2026-10-18), so either the rule is wrong or the - current Jellyfin bind path is not that token. Nothing observable settles it - today: the LDAP outpost's retained log holds only its 5-minute - `initialized direct binder` / `Update providers` config cycles and no bind - activity *(unverified 2026-10-18: no bind was performed — reproducing it - requires the bind secret)*. Re-verify with a real bind before changing the - token's intent either way. -- **Search needs `search_full_directory`.** Without the RBAC grant the - outpost returns only the bind user itself — a `(memberOf=…)` filter then - matches zero entries and the plugin logs `Found no users matching X`. - The permission is object-level on the provider; grant it via an RBAC role - and retest. -- `ak shell` import path for the provider model in 2026.8: - `from authentik.providers.ldap.models import LDAPProvider`. - *(unverified 2026-10-18: importing it outside a configured Django shell fails - with `ImproperlyConfigured`, and no `ak shell` was driven against the live - auth service for this check)* +The objects behind it — LDAP provider, bind DN, service account and bind token, +RBAC grant, groups, application slug — and the hard-won bind rules are in +[[Authentik Outpost Reference]]. ## Configuration (secrets) @@ -340,7 +204,8 @@ every outpost session (see the ORM playbook). Applications** → add an application + a **Proxy** provider whose `external_host` is exactly the browser-facing origin (`https://`), `internal_host = http://caddy:8017`, and **all three** - flows set (see the provisioning pitfall above). Attach the provider to the + flows set (see the provisioning rules in + [[Authentik Outpost Reference]]). Attach the provider to the embedded outpost. 4. For apps with native OIDC: add an **OAuth2/OpenID Connect** provider, create the app with the correct redirect URI, and paste the `client_id` / @@ -373,47 +238,6 @@ every outpost session (see the ORM playbook). the LDAP section) — bumping only the tag would leave the LDAP outpost behind. Authentik migrates the schema on startup. -## Self-serve user-settings flow (password change) - -Users change their own account details through the `default-user-settings-flow` -(`https://auth.franksdank.cloud/if/flow/default-user-settings-flow/`) — the -target of the homepage footer's "account" link. It is a `stage_configuration` -designated flow, so it renders every bound `ConfigurableStage` for the logged-in -user. - -Stages, in execution order (the `FlowStageBinding.order` field, not the M2M -insertion order): - -| order | stage | type | what it does | -|---|---|---|---| -| 20 | `default-user-settings` | `PromptStage` | name / email / locale **+ password** | -| 100 | `default-user-settings-write` | `UserWriteStage` | writes the prompt data to the user | - -**Password change is a prompt field, not a stage.** The `UserWriteStage` -writes whatever the preceding `PromptStage` collected: for the key `password` -it calls `user.set_password(value)` (`stages/user_write/stage.py`). So the fix -is a `password` field on the `default-user-settings` prompt -(`field_key=password`, `type=password`, `required=False`, order 204) — NOT a -`PasswordStage`. A `PasswordStage` is an *authentication* stage: its -`validate_password` runs `authenticate()` against the backends to check the -*current* password; it never sets a new one. (A `PasswordStage` was added on -2026-09-25 and removed the same day for exactly this reason.) - -Two ORM gotchas: `Flow.stages.add()` fails on this through model -(`Can't bulk create a multi-table inherited model` — `FlowStageBinding` is -MTI), so create the `FlowStageBinding` row directly with an explicit `order`; -and a prompt field is a `Prompt` row linked through `PromptStageField` -(`prompt_stage` + `prompt`; the order comes from `Prompt.order`). - -**A blank password must not set an empty password.** `set_password("")` would -make the account log in with no password, so a validation policy -(`default-user-settings-no-empty-password`, bound to the prompt via -`PromptStageValidationPolicy`) denies the form when `prompt_data["password"]` -is `""` and allows it when set. Verified with the `PolicyEvaluator` -(empty → deny, set → allow, absent → allow) and end-to-end by driving -`UserWriteStageView.update_user` on a throwaway user (new password valid, old -invalid). - ## Outpost claim cache (`proxysession`) — 403 after a group add The embedded outpost does not re-evaluate group membership per request. Each @@ -457,7 +281,7 @@ client → TLS edge → Anubis :8923 → outpost decision → internal (the inventory table carries both bare origins and trailing slashes; routing matches on hostname, verified 2026-09-02). `internal_host = http://caddy:8017`, all three flows set, and an - Application attached — the two provisioning rules above. + Application attached — the two provisioning rules in [[Authentik Outpost Reference]]. 2. **Route flip in ONE commit/deploy**: the vhost becomes `reverse_proxy anubis:8923` with `header_up X-Real-Ip {remote_host}`, and inside the `http://:8017` listener add `@ host ` + @@ -521,22 +345,9 @@ EOF is no `authentik.policies.group` module in this build. *(unverified 2026-10-18: these module paths were not re-executed — importing outside a configured Django shell fails with `ImproperlyConfigured`)* -- For live DB truth prefer the postgres container, and it works without any - credential — the local socket is trusted, so a read-only audit is just - `podman exec -i authentik-postgres psql -U authentik -d authentik -At -f -` - with SQL on stdin (quoting SQL through ssh+zsh otherwise fights you). Verify - before guessing: 2026.x renamed both tables and columns, and the models are - MTI all the way down — `authentik_core_provider` holds `id`, `name` and the - three `*_flow_id` columns; the proxy table is - `authentik_providers_proxy_proxyprovider` (doubled `proxy`) and its PK column - really is **`oauth2provider_ptr_id`** (proxy providers inherit the OAuth2 - table in 2026.8), the LDAP table is - `authentik_providers_ldap_ldapprovider` with PK `provider_ptr_id`; - `authentik_core_application` attaches to a provider through `provider_id` - (its PK is `policybindingmodel_ptr_id`, not `id`), and `authentik_outposts_outpost` - is keyed by `uuid`. So `Provider.objects.filter(type=…)` raising `FieldError` - is the same story from the ORM side: `type` lives in the subclass tables. - Never select secret columns (`client_secret`, `key`, `password`) into a log. +- The live database's table and column names (the 2026.x renames, the MTI parent + keys, the read-only `psql` audit recipe) are reference lookup: + [[Authentik Outpost Reference]]. ## User provisioning playbook (user-level account) diff --git a/serverAppConfigs-docs/Infrastructure/Grafana.md b/serverAppConfigs-docs/Infrastructure/Grafana.md index a1241ef..a7e6519 100644 --- a/serverAppConfigs-docs/Infrastructure/Grafana.md +++ b/serverAppConfigs-docs/Infrastructure/Grafana.md @@ -69,7 +69,7 @@ Grafana responds on `/api/health` with 200 and `"database":"ok"`; measured 2026- client_id `grafana-21ca190cde75d60f`, global authorize/token/userinfo paths, Basic client auth, form-encoded token body, and the fork's userinfo-claims bug that leaves identity at the `sub` hash only) is in - [[Authentik]] → "OAuth2 relying-party quirks". Consequences for Grafana: + [[Authentik Outpost Reference]] → "OAuth2 relying-party quirks". Consequences for Grafana: The unit sets `ALLOW_SIGN_UP=false`, `AUTO_UPSERT=true`, `ALLOW_ASSIGN_ORG=true`, `ORG_ID=1` and **no** role-attribute mapping (`grafana.container:52-55`), so users land as Viewer with `login = `; no email/groups/role claims diff --git a/serverAppConfigs-docs/Infrastructure/Homepage Status Probes.md b/serverAppConfigs-docs/Infrastructure/Homepage Status Probes.md index ac73c83..83284d8 100644 --- a/serverAppConfigs-docs/Infrastructure/Homepage Status Probes.md +++ b/serverAppConfigs-docs/Infrastructure/Homepage Status Probes.md @@ -1,7 +1,7 @@ --- type: reference title: Homepage Status Probes -description: The homepage TUI status page — how probe targets are chosen (host-netns vantage), the three false-down shapes, triage procedure, local render check, deploy, and the e2e gate interactions (asleep Mac mini, raw-IP matrix). +description: "How probe targets are chosen, the three false-down shapes, the triage procedure, and the e2e gate interactions." tags: [homepage, status, probes, e2e, homepage-status] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/Infrastructure/Homepage.md b/serverAppConfigs-docs/Infrastructure/Homepage.md index 8836544..243a56b 100644 --- a/serverAppConfigs-docs/Infrastructure/Homepage.md +++ b/serverAppConfigs-docs/Infrastructure/Homepage.md @@ -1,6 +1,6 @@ --- type: reference -description: "A password-protected web site on Dankclaw lists the services that are up, with a short description of each and a link to both the service and its usage documentation." +description: "The password-protected service list on Dankclaw: what each card shows and where its data comes from." tags: - infrastructure - homepage diff --git a/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md b/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md index e19ec35..97e9823 100644 --- a/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md +++ b/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md @@ -1,7 +1,7 @@ --- type: reference title: Observability Provisioning -description: How Prometheus config/rules and Grafana datasources/dashboards are provisioned as code in serverAppConfigs — the config triad, deploy copiers, planner mappings, seam tests, entry points for each change type, and API verification. +description: "Monitoring config as code: the config triad, deploy copiers, planner mappings, and where each change type lands." tags: [observability, prometheus, grafana, provisioning, deployment] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/Infrastructure/Observability.md b/serverAppConfigs-docs/Infrastructure/Observability.md index 2b11a80..41de35e 100644 --- a/serverAppConfigs-docs/Infrastructure/Observability.md +++ b/serverAppConfigs-docs/Infrastructure/Observability.md @@ -1,7 +1,7 @@ --- type: reference title: "Observability on Dankclaw" -description: "The Dankclaw observability stack — Prometheus, Grafana, Loki, Alloy, node-exporter and Alertmanager (Signal delivery): what runs, how it is deployed, how it is exposed, and what is still unconfigured." +description: "The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured." tags: [observability, prometheus, grafana, loki, alloy, node-exporter, quadlet, dankclaw] timestamp: 2026-09-11 --- diff --git a/serverAppConfigs-docs/Infrastructure/Prometheus.md b/serverAppConfigs-docs/Infrastructure/Prometheus.md index 31540e7..9b51a7a 100644 --- a/serverAppConfigs-docs/Infrastructure/Prometheus.md +++ b/serverAppConfigs-docs/Infrastructure/Prometheus.md @@ -1,6 +1,6 @@ --- type: reference -description: "Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints, and feeds the provisioned Grafana datasource. Down targets today are structural, not outages." +description: "Scrapes each service's metrics endpoint and feeds the provisioned Grafana datasource; down targets are structural." tags: - infrastructure - prometheus diff --git a/serverAppConfigs-docs/Infrastructure/authentik-postgres.md b/serverAppConfigs-docs/Infrastructure/authentik-postgres.md deleted file mode 100644 index f0cd9d8..0000000 --- a/serverAppConfigs-docs/Infrastructure/authentik-postgres.md +++ /dev/null @@ -1,24 +0,0 @@ ---- -type: reference -description: "The PostgreSQL 16 database for [Authentik](Authentik.md) — the homelab's SSO /" -tags: - - infrastructure - - authentik - - postgres -timestamp: 2026-08-28 ---- - -# authentik-postgres - -The PostgreSQL 16 database for [Authentik](Authentik.md) — the homelab's SSO / -OIDC provider. It is the only stateful store in the stack (users, flows, -sessions, events). Redis was removed in Authentik 2025.10; sessions and caching -now live here. - -- **Compose service:** `authentik-postgres` (in `dankclaw/compose.yaml`) -- **Image:** `docker.io/library/postgres:16-alpine` -- **Data:** `/home/dank/authentik/postgres` (bind mount; no named volumes) -- **Health:** `pg_isready -d authentik -U authentik` - -See [Authentik.md](Authentik.md) for the full architecture, setup, backups -(`pg_dump`), and failure modes. diff --git a/serverAppConfigs-docs/Infrastructure/authentik-server.md b/serverAppConfigs-docs/Infrastructure/authentik-server.md deleted file mode 100644 index e7f8255..0000000 --- a/serverAppConfigs-docs/Infrastructure/authentik-server.md +++ /dev/null @@ -1,24 +0,0 @@ ---- -type: reference -description: "The Authentik server — the core, the admin API, and the embedded Caddy outpost." -tags: - - infrastructure - - authentik -timestamp: 2026-09-16 ---- - -# authentik-server - -The Authentik server — the core, the admin API, and the embedded Caddy outpost. -This is the service the Caddy `forward_auth` vhosts proxy to -(`authentik-server:9000`). It also serves the login page at -`https://auth.franksdank.cloud`. - -- **Unit:** `dankclaw/authentik/quadlet/authentik-server.container` (`authentik-server.service`) -- **Image:** `ghcr.io/goauthentik/server:2026.8.3` (command: `server`) -- **Port:** `9000` (published; the outpost + admin + login all live here) -- **Data:** `/home/dank/authentik/data` (bind mount, shared with the worker) -- **Health:** `http://localhost:9000/-/health/` - -See [Authentik.md](Authentik.md) for the full architecture, the Caddy -forward_auth pattern, first-time setup, and failure modes. diff --git a/serverAppConfigs-docs/Infrastructure/authentik-worker.md b/serverAppConfigs-docs/Infrastructure/authentik-worker.md deleted file mode 100644 index a771b29..0000000 --- a/serverAppConfigs-docs/Infrastructure/authentik-worker.md +++ /dev/null @@ -1,23 +0,0 @@ ---- -type: reference -description: "The Authentik worker — background tasks: event processing, provider" -tags: - - infrastructure - - authentik -timestamp: 2026-09-16 ---- - -# authentik-worker - -The Authentik worker — background tasks: event processing, provider -synchronization, scheduled flows, and certificate renewal. It runs the same -image as the server; the `command: worker` selects the role. - -- **Unit:** `dankclaw/authentik/quadlet/authentik-worker.container` (`authentik-worker.service`) -- **Image:** `ghcr.io/goauthentik/server:2026.8.3` (command: `worker`) -- **Runs as:** `0:0` (rootless: container root maps to the host user that owns the data dir) -- **Data:** `/home/dank/authentik/data` (bind mount, shared with the server) -- **No port** — the worker has no HTTP endpoint. - -See [Authentik.md](Authentik.md) for the full architecture, setup, and failure -modes. diff --git a/serverAppConfigs-docs/LLM/Dankserver System Specs.md b/serverAppConfigs-docs/LLM/Dankserver System Specs.md index 53db7ff..682ffa3 100644 --- a/serverAppConfigs-docs/LLM/Dankserver System Specs.md +++ b/serverAppConfigs-docs/LLM/Dankserver System Specs.md @@ -1,7 +1,7 @@ --- -type: configuration +type: reference title: Dankserver System Specs -description: Dankserver hardware and OS state since the 2026-09-20 board swap (ASUS ProArt B850-Creator WiFi NEO, 2x AMD Radeon AI PRO R9700 gfx1201 on CPU lanes x8/x8) serving frank-medium. +description: "Dankserver hardware and OS state since the board swap: board, lane topology, cards, memory, kernel posture." tags: - dankserver - hardware diff --git a/serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md b/serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md index 5ab59d7..4eef7ca 100644 --- a/serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md +++ b/serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md @@ -1,7 +1,7 @@ --- -type: research +type: reference title: Dual R9700 Cooperation -description: Why frank-medium's two Radeon AI PRO R9700s could not run tensor-parallel on the old B650-E board (PCIe atomic routing), what the 2026-09-20 ProArt B850-Creator swap unblocked, and what still has to be probed. +description: "Why two R9700s could not run tensor-parallel on the old board, and what the board swap unblocked." tags: - frank-medium - rocm diff --git a/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md b/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md index df516ab..e1a4edc 100644 --- a/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md +++ b/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md @@ -1,7 +1,7 @@ --- -type: research +type: reference title: FastFlowLM NPU Inference -description: FastFlowLM (FLM), the AMD NPU-first LLM runtime for Ryzen AI XDNA2 chips. Fit analysis for the 50 TOPS NPU on Dankclaw, published benchmarks, verified install state, and a test runbook. +description: "The AMD NPU-first runtime: fit analysis for the XDNA2, published benchmarks, install state, and test runbook." tags: - fastflowlm - npu diff --git a/serverAppConfigs-docs/LLM/FreeToken Builds.md b/serverAppConfigs-docs/LLM/FreeToken Builds.md index 3120581..87caf65 100644 --- a/serverAppConfigs-docs/LLM/FreeToken Builds.md +++ b/serverAppConfigs-docs/LLM/FreeToken Builds.md @@ -1,7 +1,7 @@ --- type: reference title: FreeToken Builds -description: Building FreeToken (the official franksdank fork of FreeTok Tokenworks' freetoken vLLM-MTP runtime) on Dankclaw — the CUDA vLLM venv build and the ROCm build, with the dependency traps each one hit. +description: "Building the FreeToken runtime here: the CUDA and ROCm builds and the dependency trap each one hit." tags: [freetoken, build, dankclaw, vllm, mtp] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/LLM/GGUF Compatibility.md b/serverAppConfigs-docs/LLM/GGUF Compatibility.md index aae7866..f66f42e 100644 --- a/serverAppConfigs-docs/LLM/GGUF Compatibility.md +++ b/serverAppConfigs-docs/LLM/GGUF Compatibility.md @@ -1,7 +1,7 @@ --- type: reference title: GGUF Compatibility and Runtime Lock-in -description: How to answer "will engine X load this GGUF?" from the file's header alone, why a custom tensor type forces the runtime, and the discipline for pinning a fork build that can actually serve the file. +description: "Answering whether an engine loads a GGUF from the file header alone, and pinning a fork build that can." tags: [gguf, llama.cpp, forks, quantization, speculative] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/LLM/Heterogeneous Split Triage.md b/serverAppConfigs-docs/LLM/Heterogeneous Split Triage.md index 611dbad..9952905 100644 --- a/serverAppConfigs-docs/LLM/Heterogeneous Split Triage.md +++ b/serverAppConfigs-docs/LLM/Heterogeneous Split Triage.md @@ -1,7 +1,7 @@ --- type: reference title: Heterogeneous Split Triage -description: Triage before building when someone asks to split a model across unequal accelerators on one host (24 GB GDDR6X + LPDDR APU, PCIe x4 link) — link characterisation, the real bottleneck, what placement never buys, UMA pools, drafter splits, and the verdict ranking. +description: "Triage before splitting across unequal accelerators: link characterisation, real bottleneck, what placement never buys." tags: [heterogeneous, multi-device, llama-cpp, triage, pcie] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md b/serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md index 87385cd..1668a50 100644 --- a/serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md +++ b/serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md @@ -1,7 +1,7 @@ --- type: reference title: LLM Bench Windows and Measurement -description: Stopping-and-restoring benchmark windows on the fleet LLM hosts (llama-bench sweeps, HTTP param benches, multi-arm vLLM offload sweeps), honest benchmark methodology, and the usage-counter measurement law under speculative decoding. +description: "Opening and restoring benchmark windows on fleet hosts, honest methodology, and the usage-counter law under drafting." tags: [benchmarking, llama-bench, ab-testing, speculative, vllm, fleet] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md b/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md index 27a0e17..7ccd078 100644 --- a/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md +++ b/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md @@ -1,7 +1,7 @@ --- type: reference title: LLM Performance Diagnosis -description: Probing why a served LLM is slow — the flat-depth probe, the expert-offload duty-cycle arithmetic, kernel-time attribution via the profiler window, live telemetry without benchmarking, RAM-bandwidth ceilings, IBS tensor attribution and prefix-cache-aware routing. +description: "Probing why a served model is slow: the flat-depth probe, offload duty-cycle arithmetic, and profiler attribution." tags: [performance, profiling, moe, pcie, prefix-cache, routing, telemetry] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/LLM/Llama Cpp Multi-Device Splits.md b/serverAppConfigs-docs/LLM/Llama Cpp Multi-Device Splits.md index f97e322..896f0ab 100644 --- a/serverAppConfigs-docs/LLM/Llama Cpp Multi-Device Splits.md +++ b/serverAppConfigs-docs/LLM/Llama Cpp Multi-Device Splits.md @@ -1,7 +1,7 @@ --- type: reference title: Llama Cpp Multi-Device Splits -description: What llama.cpp can and cannot express across devices, the no-collective layer-split path, split balancing by measurement, drafter KV sizing from GGUF headers, and heterogeneous placement triage. +description: "What llama.cpp can express across devices: the no-collective layer-split path, balancing, and drafter KV sizing." tags: [llama.cpp, multi-gpu, layer-split, pcie, sizing, speculative] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/LLM/Llama Cpp Serving Sizing.md b/serverAppConfigs-docs/LLM/Llama Cpp Serving Sizing.md index 28c1c8a..f427e68 100644 --- a/serverAppConfigs-docs/LLM/Llama Cpp Serving Sizing.md +++ b/serverAppConfigs-docs/LLM/Llama Cpp Serving Sizing.md @@ -1,7 +1,7 @@ --- type: reference title: Llama Cpp Serving Sizing -description: Slot-pool sizing against a headroom band, per-device VRAM accounting, prompt-cache/state sizing and the QSA flash-attn fix — the measurement-first procedures for llama.cpp serving shapes. +description: "Measurement-first sizing for llama.cpp shapes: slot pools against a headroom band, per-device accounting, prompt cache." tags: [llama.cpp, vram, kv-cache, slots, flash-attn, sizing] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/LLM/Model Swap and Rollback.md b/serverAppConfigs-docs/LLM/Model Swap and Rollback.md index 74c49bd..3b4fdad 100644 --- a/serverAppConfigs-docs/LLM/Model Swap and Rollback.md +++ b/serverAppConfigs-docs/LLM/Model Swap and Rollback.md @@ -1,7 +1,7 @@ --- type: reference title: Model Swap and Rollback -description: The full model-swap discipline for the LLM fleet — fit preflight, fetch and payload verification, runtime build/flag checks, git-only config, live validation, systemd handoff, close-out, the three-pin revert law, and research-verification discipline. +description: "The model-swap discipline: fit preflight, fetch verification, build and flag checks, live validation, three-pin revert." tags: [models, swap, gguf, huggingface, rollback, deployment, verification] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/LLM/PCIe Link and Atomics.md b/serverAppConfigs-docs/LLM/PCIe Link and Atomics.md index 37dcfa3..7d4ce3c 100644 --- a/serverAppConfigs-docs/LLM/PCIe Link and Atomics.md +++ b/serverAppConfigs-docs/LLM/PCIe Link and Atomics.md @@ -1,7 +1,7 @@ --- type: reference title: PCIe Link and Atomics -description: Link probing (idle downshift, path ceilings, CUDA/HIP bandwidth probes), the streaming-vs-compute ceiling arithmetic, and the PCIe-atomics wall that decides whether AMD multi-GPU collectives can exist at all. +description: "Link probing, the streaming-versus-compute ceiling arithmetic, and the atomics wall that decides multi-GPU collectives." tags: [pcie, rccl, atomics, multi-gpu, bandwidth, amd, nvidia] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Ethernet Bandwidth Math.md b/serverAppConfigs-docs/LLM/Pipeline Parallelism/Ethernet Bandwidth Math.md index 9044dd1..36fc071 100644 --- a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Ethernet Bandwidth Math.md +++ b/serverAppConfigs-docs/LLM/Pipeline Parallelism/Ethernet Bandwidth Math.md @@ -1,11 +1,11 @@ --- type: reference title: Ethernet Bandwidth Math -description: Bytes-per-token math for pipeline vs tensor parallelism, feasibility tables for 1/2.5/10 GbE, and real multi-node benchmark numbers. +description: "Bytes-per-token math for pipeline versus tensor parallelism, and the link budgets that follow from it." tags: - pipeline-parallelism - research -timestamp: 2026-08-17 +timestamp: 2026-10-19 --- # Bandwidth Analysis: Multi-Node LLM Inference over Ethernet @@ -147,19 +147,11 @@ bytes_per_boundary = batch_size × hidden_size × 2 [fp16 = 2 bytes] - Per-token, per-boundary: `6144 × 2 = 12,288 bytes = 12 KB` - With Nₚₚ stage boundaries: total = `Nₚₚ × 12 KB/tok` -**Full example — 27B model, 2 PP stages (1 boundary), batch=1:** +**Worked examples (1 boundary = 2 PP stages):** ``` -1 boundary × 4096 × 2 bytes = 8,192 bytes/tok -``` - -**Example — same model, batch=64:** -``` -1 boundary × 4096 × 64 × 2 bytes = 524,288 bytes = 512 KB/tok -``` - -**Example — 36B model, 2 PP stages, batch=1:** -``` -1 boundary × 6144 × 2 bytes = 12,288 bytes = 12 KB/tok +27B, batch=1: 1 boundary × 4096 × 2 bytes = 8,192 bytes/tok = 8 KB +27B, batch=64: 1 boundary × 4096 × 64 × 2 bytes = 524,288 = 512 KB/tok +36B, batch=1: 1 boundary × 6144 × 2 bytes = 12,288 bytes/tok = 12 KB ``` ### 1.2 Tensor Parallelism (TP) — for comparison @@ -244,57 +236,13 @@ total_bytes_per_tok = L × batch_size × hidden_size × 2 [L all-reduces total | 50 | 10.24 Mbit/s | 163.84 Mbit/s | 655.36 Mbit/s | | 100 | 20.48 Mbit/s | 327.68 Mbit/s | 1,310.72 Mbit/s | -### 2.3 Feasibility matrix +### 2.3 Feasibility -Green = feasible (< 50% of link capacity). Yellow = marginal (50–90%). Red = saturated (> 90%). - -**27B model — PP (1 boundary), batch=1:** - -| | 1 GbE (118 MB/s) | 2.5 GbE (294 MB/s) | 10 GbE (1175 MB/s) | -|---|---|---|---| -| 10 tok/s | ✅ 0.13 Mbit/s | ✅ 0.13 Mbit/s | ✅ 0.13 Mbit/s | -| 50 tok/s | ✅ 0.65 Mbit/s | ✅ 0.65 Mbit/s | ✅ 0.65 Mbit/s | -| 100 tok/s | ✅ 1.31 Mbit/s | ✅ 1.31 Mbit/s | ✅ 1.31 Mbit/s | - -**27B model — PP (1 boundary), batch=16:** - -| | 1 GbE | 2.5 GbE | 10 GbE | -|---|---|---|---| -| 10 tok/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s | -| 50 tok/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s | -| 100 tok/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s | - -**27B model — PP (1 boundary), batch=64:** - -| | 1 GbE | 2.5 GbE | 10 GbE | -|---|---|---|---| -| 10 tok/s | ✅ 8.19 Mbit/s | ✅ 8.19 Mbit/s | ✅ 8.19 Mbit/s | -| 50 tok/s | ✅ 40.96 Mbit/s | ✅ 40.96 Mbit/s | ✅ 40.96 Mbit/s | -| 100 tok/s | ✅ 81.92 Mbit/s | ✅ 81.92 Mbit/s | ✅ 81.92 Mbit/s | - -**27B model — TP (32 layers), batch=1:** - -| | 1 GbE | 2.5 GbE | 10 GbE | -|---|---|---|---| -| 10 tok/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s | -| 50 tok/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s | -| 100 tok/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s | - -**27B model — TP (32 layers), batch=16:** - -| | 1 GbE | 2.5 GbE | 10 GbE | -|---|---|---|---| -| 10 tok/s | ✅ 32.77 Mbit/s | ✅ 32.77 Mbit/s | ✅ 32.77 Mbit/s | -| 50 tok/s | ✅ 163.84 Mbit/s | ✅ 163.84 Mbit/s | ✅ 163.84 Mbit/s | -| 100 tok/s | ✅ 327.68 Mbit/s | ✅ 327.68 Mbit/s | ✅ 327.68 Mbit/s | - -**27B model — TP (32 layers), batch=64:** - -| | 1 GbE | 2.5 GbE | 10 GbE | -|---|---|---|---| -| 10 tok/s | ✅ 131.07 Mbit/s | ✅ 131.07 Mbit/s | ✅ 131.07 Mbit/s | -| 50 tok/s | ✅ 655.36 Mbit/s | ✅ 655.36 Mbit/s | ✅ 655.36 Mbit/s | -| 100 tok/s | ✅ 1310.72 Mbit/s | ✅ 1310.72 Mbit/s | ✅ 1310.72 Mbit/s | +Thresholds: **feasible** = under 50% of link capacity, **marginal** = 50–90%, +**saturated** = over 90%. Cross-checking §2.2 against §2.1: every PP case is +feasible on every link class — even the heaviest, 27B at batch=64 and 100 +tok/s (81.92 Mbit/s), is feasible on 1 GbE — and every TP case is feasible +except batch=64: **27B TP batch=64 at 100 tok/s: 1310.72 Mbit/s > 1 GbE usable (1000 Mbit/s) — link-saturated on 1 GbE** @@ -368,10 +316,7 @@ With TP, each layer transfers `H × 2` bytes but this happens L times. For 256 K **Petals [1]:** - Uses pipeline parallelism with dynamic 8-bit quantization (halves bandwidth) - For BLOOM-176B (H=14336), per-boundary communication is ~14 KB/tok with quantization -- Recommended bandwidth: 25 Mbit/s bidirectional for interactive inference [1] - Key finding: "For inference, performance does not depend much on bandwidth or sequence length but degrades with higher latency" — RTT matters more than throughput -- At 1 Gbit/s with <5 ms RTT: 1.71 steps/s; at 100 Mbit/s with <5 ms: 1.66 steps/s (only 3% difference) -- At 100 Mbit/s with 100 ms RTT: 1.23 steps/s (28% degradation) **llama.cpp RPC [2]:** - Model weights and KV cache distributed across nodes proportionally to memory @@ -493,38 +438,17 @@ With Nₚₚ = 4 and 30 µs per-hop RTT: ~120 µs network, still <6% of compute ## Appendix: Full bandwidth formulae -### Pipeline Parallelism (generation, batch=1) +### Pipeline and Tensor Parallelism (generation) + +`N` is the batch size; `N = 1` is the batch=1 form. ``` -bytes_per_boundary_per_tok = hidden_size × 2 [fp16] -total_bytes_per_tok = num_boundaries × hidden_size × 2 +bytes_per_boundary_per_tok = hidden_size × N × 2 [fp16, PP boundary] +bytes_per_layer_allreduce = hidden_size × N × 2 [TP, one all-reduce per layer] +total_bytes_per_tok = num_boundaries × hidden_size × N × 2 [PP] +total_bytes_per_tok = num_layers × hidden_size × N × 2 [TP] bandwidth_mbps = total_bytes_per_tok × tokens_per_second × 8 / 1,000,000 -``` - -### Pipeline Parallelism (generation, batch=N) -``` -bytes_per_boundary_per_tok = hidden_size × N × 2 -total_bytes_per_tok = num_boundaries × hidden_size × N × 2 -``` - -### Tensor Parallelism (generation, batch=1) - -``` -bytes_per_layer_allreduce = hidden_size × 2 -total_bytes_per_tok = num_layers × hidden_size × 2 -bandwidth_mbps = total_bytes_per_tok × tokens_per_second × 8 / 1,000,000 -``` - -### Tensor Parallelism (generation, batch=N) - -``` -total_bytes_per_tok = num_layers × hidden_size × N × 2 -``` - -### With 8-bit quantization (like Petals) - -``` -# Simply divide all fp16 byte counts by 2 -bytes_per_boundary_per_tok = hidden_size × 1 [int8] +# With 8-bit quantization (like Petals): simply divide all fp16 byte counts by 2 +bytes_per_boundary_per_tok = hidden_size × 1 [int8] ``` diff --git a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Expected Performance.md b/serverAppConfigs-docs/LLM/Pipeline Parallelism/Expected Performance.md index 7ebb709..4fc57e2 100644 --- a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Expected Performance.md +++ b/serverAppConfigs-docs/LLM/Pipeline Parallelism/Expected Performance.md @@ -1,7 +1,7 @@ --- type: reference title: Expected Performance -description: Estimated tokens per second for a 25-40B model split across two nodes over 1/2.5/10 GbE, with real benchmark anchors. +description: "Estimated split-model throughput over ethernet, with the benchmark anchors the estimates rest on." tags: - pipeline-parallelism - performance @@ -13,7 +13,7 @@ timestamp: 2026-08-14 This page estimates the speed of a 25-40B model split across two machines over ethernet. It uses the bandwidth math in [[Ethernet Bandwidth Math]] -and the measured numbers in [[Mixed-Vendor Experiments]]. +and the mixed-vendor measurements in [[Multi-Node Runtimes]]. ## The Bandwidth Ceiling Is Not the Limit @@ -93,5 +93,5 @@ roughly 200k-260k tokens across both nodes. ## See Also - [[Ethernet Bandwidth Math]] -- [[Mixed-Vendor Experiments]] +- [[Multi-Node Runtimes]] - [[Sample Script Set]] diff --git a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Mixed-Vendor Experiments.md b/serverAppConfigs-docs/LLM/Pipeline Parallelism/Mixed-Vendor Experiments.md deleted file mode 100644 index f04ce78..0000000 --- a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Mixed-Vendor Experiments.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: reference -title: "Mixed-Vendor Experiments" -description: "Real-world NVIDIA + AMD Strix Halo multi-node experiments: what works (llama.cpp RPC, Vulkan split), what breaks (mixed CUDA+ROCm vLLM), measured tok/s." -tags: - - pipeline-parallelism - - research -timestamp: 2026-08-14 ---- diff --git a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Multi-Node Runtimes.md b/serverAppConfigs-docs/LLM/Pipeline Parallelism/Multi-Node Runtimes.md new file mode 100644 index 0000000..b99a4e6 --- /dev/null +++ b/serverAppConfigs-docs/LLM/Pipeline Parallelism/Multi-Node Runtimes.md @@ -0,0 +1,257 @@ +--- +type: reference +title: Multi-Node Runtimes +description: "How each two-machine runtime is built and launched, what it can combine with, and the mixed-vendor verdicts." +tags: + - pipeline-parallelism + - llamacpp + - vllm + - speculative +timestamp: 2026-10-19 +--- + +# Multi-Node Runtimes + +Two runtimes can put one model on two machines: llama.cpp over the ggml-rpc +backend, and vLLM with pipeline parallelism. This page owns the cross-runtime +facts — how each is built and launched, what each can and cannot be combined +with, and the mixed-vendor verdicts. Facts owned by a sibling page are linked, +not restated. Bytes-per-token arithmetic and link budgets belong to +[[Ethernet Bandwidth Math]], throughput estimates to [[Expected Performance]], +the real deployment and its verified gotchas to [[Qwen3.8 PP Deployment]], the +educational scripts to [[Sample Script Set]], drafter mechanisms to +[[DFlash DSpark Speculative Decoding]], single-host multi-GPU placement to +[[Llama Cpp Multi-Device Splits]]. + +## What each runtime is + +Both distribute by **layers**, not inside layers. + +| | llama.cpp over ggml-rpc | vLLM pipeline parallelism | +| --- | --- | --- | +| unit of distribution | a contiguous slice of layers (`--split-mode layer`, the default) | a contiguous slice of layers (`--pipeline-parallel-size`) | +| process model | one `llama-server`/`llama-cli` on the head; a `ggml-rpc-server` per worker | one `vllm serve` per node, one engine across them | +| what crosses the wire | whatever graph work the head offloads to the remote devices | the hidden state at each stage boundary, point to point | +| KV cache | distributed with the weights; the KV of a layer lives with the owner of that layer | each stage holds the KV of the layers it owns | +| speculation together with the split | yes in llama.cpp (see below) | no, per vLLM's own docs | + +Upstream's own summary is why layer-wise is the ethernet choice: pipeline +parallel "minimizes data transfers between GPUs but requires many tokens to +scale well", while tensor parallel "does multiple cross-GPU reductions per +layer … much more bottlenecked by the GPU interconnect speed". + +Do not carry the tensor path across machines: `--split-mode tensor` is +experimental, needs flash attention, rejects quantized KV, and is not +implemented for the MoE and hybrid architectures the guide lists — and +per-tensor placement across vendors is not expressible (§1 of [[Llama Cpp Multi-Device Splits]]). + +## llama.cpp over ggml-rpc + +### Build and launch + +Both sides take `-DGGML_RPC=ON` next to that host's own backend: + +```bash +cmake .. -DGGML_CUDA=ON -DGGML_RPC=ON # a CUDA worker; -DGGML_HIP=ON for an AMD one +cmake --build . --config Release +``` + +Worker side — `ggml-rpc-server` exposes ggml devices on the remote host: + +```bash +bin/ggml-rpc-server # endpoint 127.0.0.1:50052 by default +bin/ggml-rpc-server --device CUDA0 -c # or CUDA_VISIBLE_DEVICES=0; -c = local cache +``` + +- It exposes every accelerator it can see; a host with no accelerator exposes a + single `CPU` device. `--device` and `CUDA_VISIBLE_DEVICES` are equivalent + ways to narrow that set. +- `-c` keeps large tensors in `$HOME/.cache/llama.cpp/rpc` (redirect with + `LLAMA_CACHE`) so a big model does not re-stream per load; that cache + filling the worker disk is a recorded fleet incident ([[Qwen3.8 PP Deployment]]). + +Head side — build the local backends plus `-DGGML_RPC=ON`, then name the +workers: + +```bash +llama-server -m model.gguf --rpc 192.168.88.10:50052,192.168.88.11:50052 -ngl 99 +``` + +`--rpc` is a comma-separated list of `host:port`, one entry per +`ggml-rpc-server`; the head's own devices stay in the pool next to the remote +ones. Launch scripts for this shape: [[Sample Script Set]]. + +### Split, context, KV + +- Default placement: weights **and** KV across all devices, local and remote, + "in proportion to each device's available memory". Override with + `--tensor-split`, whose proportions follow the order in `--device`. Why the + fleet always pins it (a Strix reports a huge GTT window, so the automatic + split lands on the slow device) is [[Qwen3.8 PP Deployment]]'s. +- `-sm layer` (the default) is pipeline parallelism: each device holds a + contiguous slice of layers and the KV cache of a layer lives on the device + that owns that layer. +- KV types are the usual `-ctk` / `-ctv`, `f16` by default. `--fit` is on by + default and sizes unset arguments to device memory; turn it off to hold a + context you set on purpose. Context is one global `--ctx-size` — the RPC + README documents no per-node context or per-node KV knob. +- Upstream's out-of-memory ladder (stated for tensor mode, useful order + anyway): lower `--ctx-size`, then `--parallel`, then `--n-gpu-layers`. +- Transport is TCP; RDMA is negotiated at the handshake with no flag changes + and falls back to TCP unless both peers can use it (Linux RoCEv2 via + `libibverbs`, macOS RDMA over Thunderbolt 5 via `librdma`). It is point to + point — name the peer's RDMA-capable address in `--rpc` or the session stays + on TCP. `GGML_RPC_NO_RDMA=1` forces TCP; `GGML_RPC_DEBUG` traces the backend. + +Upstream calls the backend a proof of concept, "fragile and insecure" (see the +Security section). Two limits shape the design: decode is bound by the per-step +round trip, not by the link's bytes ([[Ethernet Bandwidth Math]]), and every +node must run the identical build or the handshake hangs ([[Qwen3.8 PP Deployment]]). + +## vLLM pipeline parallelism + +> **Provenance.** Flags and mechanics below were recorded 2026-08-14 and are +> not re-checked upstream in this pass (see "Unverified in this pass"); the +> incompatibility lines, the method catalog and the issue history are re-verified. + +- `--pipeline-parallel-size` divides the layers into contiguous stages; + `--tensor-parallel-size` still works inside a node, so the canonical two-node + shape is tp = GPUs per node, pp = nodes. +- Two backends: Ray (`--distributed-executor-backend ray`, the worker joining + a `ray start --head`) or multiprocessing (`--distributed-executor-backend mp` + with `--nnodes`, `--node-rank`, `--master-addr`, `--headless` on the + follower). `mp` avoids the Ray dependency; Ray brings the cluster manager. +- What the interconnect must carry: one activation tensor per stage boundary + per forward pass, point to point — not a per-layer reduction. That is the + whole cross-node traffic story; the byte arithmetic and link budgets are + [[Ethernet Bandwidth Math]]. Pin the NIC vLLM talks on + (`NCCL_SOCKET_IFNAME`) rather than letting it guess an interface. +- The PP bubble: with N stages and one sequence in flight, one stage works + while N-1 idle, so pp=2 costs half the machine at batch one. Batching is the + cure — pipeline parallelism "requires many tokens to scale well" — and the + microbatch formula is [[Ethernet Bandwidth Math]] §5.3. +- KV and context: each stage keeps the KV of its own layers, so the pool grows + with the stage count, and `--max-model-len` is not reduced by the split + (recorded 2026-08-14, unverified here). +- What it cannot combine with, in vLLM's words: "Pipeline parallelism is not + composable with speculative decoding as of `vllm<=0.15.0`", and "Speculative + decoding with a draft models is not supported in `vllm<=0.10.0`". Composing + the two has been requested since 2024 and closed stale (#6911, #10615); PP + also carries a Qwen3.5 bug report closed stale without a fix (#36643). +- Method catalog at v0.25.1: EAGLE, MTP, draft model, PARD, MLP, n-gram, suffix + decoding, hidden-state extraction, a custom proposer class, dynamic + speculative decoding; `dflash` is an accepted `method` value. + +## Speculative decoding across the split + +llama.cpp chains several implementations behind one comma-separated +`--spec-type` list; the split does not change the chain, only where its passes +run. Mechanisms, chaining laws and drafter KV sizing are +[[DFlash DSpark Speculative Decoding]]'s; only the split interaction is here. + +| Method | llama.cpp type | Over ggml-rpc | With vLLM PP | Why | +| --- | --- | --- | --- | --- | +| MTP | `draft-mtp` | works; not documented upstream, but run this way on this fleet's split head with acceptance measured ([[Qwen3.8 PP Deployment]]) | no | Context-backed drafter: it needs a draft context and runs in the head process, so RPC devices are simply more devices under it | +| DFlash | `draft-dflash` | same gate, same status as MTP | no | Same draft-context gate (`ctx_dft`); the drafter's KV lands on whichever device the draft context is built on — the one thing to check on a split | +| ngram-mod | `ngram-mod` | safest of the four: no draft model, no draft context, no remote draft pass | no | Draftless — proposals come from cached token n-grams in the head, so the split adds no device work | +| DSpark | `draft-dspark` | **unproven over RPC** — not a missing feature | no | `draft-dspark` exists upstream; nobody has measured it with the target split across two machines | + +**Correction (2026-10-19).** A retired stub in this directory asserted "DSpark +is vLLM-only." False. `draft-dspark` is in upstream llama.cpp — parsed in +`common/arg.cpp`, mapped to `COMMON_SPECULATIVE_TYPE_DRAFT_DSPARK` in +`common/speculative.cpp`, where it carries a Markov head and an anchor-first +block layout and aborts with `DSpark draft has no confidence head: please set +--spec-draft-p-min 0` when the drafter has no confidence head. What is unproven +is running it over RPC, and that is the honest limit of the claim. + +The vLLM column is uniform because that statement is blanket: it names pipeline +parallelism, not a method, so n-gram and suffix are excluded as written. vLLM +gives no mechanism with it; "the drafter must sit next to the target's layers, +which a stage split separates" is a reading, not upstream's words. + +Upstream likewise documents neither for nor against `--rpc` plus +`--spec-type`. What the implementation does say is structural: every +speculative implementation is built in the head process against the head's +device set, and draftless types need no draft context at all. Routing draft +passes to a remote worker, however, has never been tested. + +## Mixed-vendor reality + +What works today: + +- **One vendor per node.** RPC is backend-agnostic by construction: upstream's + own diagram has a host exposing CUDA0+CUDA1, a host exposing Metal, and a + host exposing CUDA0+CPU, all driven by one main host over TCP. Each node + builds its own backend plus `-DGGML_RPC=ON`; nothing speaks CUDA-to-ROCm. +- **Identical build tag on every node**, worker restarted before head. The + verified version-skew gotcha is [[Qwen3.8 PP Deployment]]'s. +- **Speculation on the head** while the split carries the target (table above). +- **RDMA when both peers have it**, auto-negotiated at the handshake; the size + of the gain is third-party and unverified here. + +Unproven or broken: + +- **One binary holding both CUDA and ROCm for a same-host layer split**: + reported to compile and then list only CUDA devices (issue #11506, + third-party, unverified). Vulkan is the reported single-binary cross-vendor + path, and single-host placement is [[Llama Cpp Multi-Device Splits]]'s, whose + verdict is that per-tensor placement across a vendor mix never works. +- **vLLM across a CUDA node and a ROCm node**: no known experiment; both ends + run the same backend (recorded 2026-08-14). AMD's own two-Halo clustering + guide is tensor parallelism over RCCL, not a mixed-vendor split. +- **DSpark over RPC**: unproven, see the correction above. +- **The exact NVIDIA-desktop + Strix-Halo-NUC pair has no published + benchmark**, and the third-party mixed-vendor figures it is estimated from + (Metal+CUDA over 10 GbE, two Strix Halos over USB4, a three-card CUDA+ROCm set + over 1 GbE, a Vulkan triple in one box, an AMD four-node Strix cluster) live on + [[Expected Performance]] and [[Ethernet Bandwidth Math]], each named with its + model and hardware. None was re-verified in this pass. + +## Security + +`ggml-rpc-server` has **no authentication and no encryption**. Upstream's own +warning: the backend is "in a proof-of-concept development stage. As such, the +functionality is fragile and insecure. Never run the RPC server on an open +network or in a sensitive environment!" It binds `127.0.0.1:50052` by default — +safe by accident. Bind a routable address and any host that can reach the port +can allocate memory there and supply the tensors it computes on. Trusted LAN +only, firewalled or tunnelled; never a public address. + +## Unverified in this pass + +Every item was first recorded 2026-08-14, in the research that preceded this +page. Nothing is silently deleted; none is backed by the sources cited below. + +- **RPC protocol version and handshake rule** — recorded as protocol `5.0.0` + with "major must match, server minor must be ≥ client". The README's own + banner prints `Starting RPC server v3.0.0`; the conflict is unresolved. +- **RPC internals** — a hard cap of 16 servers, synchronous graph compute, no + tensor copy between two RPC servers, workers loading tensors on first use, a + hash-keyed cache above roughly 10 MB, asserts on quantized tensors whose rows + are not 512-aligned, and `-rpc` on `llama-bench` (the README names only + `llama-cli` and `llama-server`). +- **vLLM flag surface** — `--pipeline-parallel-size`, `--tensor-parallel-size`, + `--distributed-executor-backend {ray,mp}`, `--nnodes`, `--node-rank`, + `--master-addr`, `--headless`, the `ray start --head` join, + `NCCL_SOCKET_IFNAME`. +- **vLLM mechanics** — one activation handoff per stage boundary per forward + pass, a worked ~512 MB handoff for a hidden-5120 model at batch 8 and 2048 + positions, `--max-model-len` unchanged by PP, ~50% idle at pp=2 batch one. +- **vLLM throughput verdicts** for a 27B across two nodes over gigabit-class + links; superseded as an estimate by [[Expected Performance]]. +- **Mixed-vendor claims** — a CUDA+HIP build listing only CUDA devices + (#11506), Vulkan as the universal cross-vendor backend, AMD's two-Halo + clustering guide being RCCL tensor parallelism, WiFi costing about ten times + wired, the slowest hop setting the decode pace, RDMA worth a few percent. +- **Draft over RPC** — that the draft model stays on the head, and that routing + draft passes to a remote worker has never been confirmed. + +## Citations + +- [llama.cpp RPC README](https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md), accessed 2026-10-19 — server usage and device exposure, `-DGGML_RPC=ON` on both sides, the `--rpc host:port` list, proportional weight+KV split and `--tensor-split`, the `-c` local cache, RDMA negotiation and `GGML_RPC_NO_RDMA`, the proof-of-concept and no-authentication warning. +- [llama.cpp multi-GPU guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md), accessed 2026-10-19 — split modes (`-sm layer` = pipeline parallelism, KV with the layer owner), `-ts`/`-ngl`/`-ctk`/`-ctv`/`-fit`, the pipeline-versus-tensor summary, the tensor-mode limits. +- [vLLM speculative decoding docs, v0.25.1](https://docs.vllm.ai/en/v0.25.1/features/speculative_decoding/), accessed 2026-10-19 — the pipeline-parallelism incompatibility statement, the draft-model version floor, the method catalog. +- [llama.cpp `common/speculative.cpp`](https://github.com/ggml-org/llama.cpp/blob/master/common/speculative.cpp), accessed 2026-10-19 — the speculative implementation set including `draft-dspark`, its draft-context gate, its confidence-head error. +- vLLM issue #6911, "[Feature]: Combine pipeline parallelism with speculative decoding", opened 2024-07-29, closed not-planned 2025-03-28, accessed 2026-10-19. +- vLLM issue #10615, "[Usage]: Does speculative decoding support pipeline parallelism?", opened 2024-11-25, closed not-planned 2025-04-18, accessed 2026-10-19. +- vLLM issue #36643, "[Bug]: Qwen3.5 does not work with pipeline parallelism", opened 2026-03-10, closed not-planned 2026-09-24, accessed 2026-10-19. diff --git a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.6 27B Spec Methods.md b/serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.6 27B Spec Methods.md deleted file mode 100644 index d991baf..0000000 --- a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.6 27B Spec Methods.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: reference -title: "Qwen3.6 27B Spec Methods" -description: "Qwen3.6-27B Q8 GGUF variants and llama.cpp speculative decoding (draft-mtp, draft-dflash, ngram-mod): flags, RPC compatibility, memory/context math." -tags: - - pipeline-parallelism - - research -timestamp: 2026-08-14 ---- diff --git a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.8 PP Deployment.md b/serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.8 PP Deployment.md index 4fb30f6..47133b7 100644 --- a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.8 PP Deployment.md +++ b/serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.8 PP Deployment.md @@ -1,6 +1,6 @@ --- type: reference -description: "Split Qwen3.8-27B across the Strix Halo iGPU (head) and the RTX" +description: "The real split deployment: which node is head and which is worker, the flags, and the verified gotchas." tags: - llm - pipeline diff --git a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Sample Script Set.md b/serverAppConfigs-docs/LLM/Pipeline Parallelism/Sample Script Set.md index 3fc7c9f..24313f6 100644 --- a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Sample Script Set.md +++ b/serverAppConfigs-docs/LLM/Pipeline Parallelism/Sample Script Set.md @@ -1,7 +1,7 @@ --- type: reference title: Sample Script Set -description: An educational two-node llama.cpp RPC split of Qwen3.6-27B-Q8_K_P over ethernet. For reading only. Do not run or deploy. +description: "An educational two-node RPC split script set, for reading only — never run it or deploy it." tags: - pipeline-parallelism - sample @@ -255,7 +255,7 @@ is 30 GiB. ## See Also -- [[llama.cpp RPC]] +- [[Multi-Node Runtimes]] - [[Expected Performance]] - [[Ethernet Bandwidth Math]] -- [[Qwen3.6 27B Spec Methods]] \ No newline at end of file +- [[DFlash DSpark Speculative Decoding]] \ No newline at end of file diff --git a/serverAppConfigs-docs/LLM/Pipeline Parallelism/index.md b/serverAppConfigs-docs/LLM/Pipeline Parallelism/index.md index e3ccc16..9e7f00f 100644 --- a/serverAppConfigs-docs/LLM/Pipeline Parallelism/index.md +++ b/serverAppConfigs-docs/LLM/Pipeline Parallelism/index.md @@ -28,28 +28,24 @@ graph LR end ``` -- [[llama.cpp RPC]] — the working, production-capable path for two heterogeneous nodes over ethernet. -- [[vLLM Pipeline Parallelism]] — supported but heavier; needs Ray or multiprocessing and does not compose with speculative decoding yet. ## What the Research Found -- [[Ethernet Bandwidth Math]] — a PP boundary moves only the hidden state +- [[Ethernet Bandwidth Math]] — Bytes-per-token math for pipeline versus tensor parallelism, and the link budgets that follow from it. (about 10 KiB per token for a 27B model). 1 GbE supports hundreds of tokens per second on paper; the real limit is latency and RPC overhead. -- [[Mixed-Vendor Experiments]] — NVIDIA + Strix Halo multi-node inference works today via llama.cpp RPC with separate CUDA and ROCm builds. The exact NVIDIA-desktop + Strix-Halo-NUC pair has not been benchmarked with published numbers yet. -- [[Qwen3.6 27B Spec Methods]] — Q8 GGUF variants, and which speculative methods (MTP, DFlash, ngram-mod) work with RPC. DSpark is vLLM-only. -- [[Expected Performance]] — estimated tokens per second for a 27B Q8 split +- [[Expected Performance]] — Estimated split-model throughput over ethernet, with the benchmark anchors the estimates rest on. across an RTX 3090 and a Strix Halo NUC over 1/2.5/10 GbE. ## Deployment -- [[Qwen3.8 PP Deployment]] — the real Qwen3.8-27B split: head on Dankclaw +- [[Qwen3.8 PP Deployment]] — The real split deployment: which node is head and which is worker, the flags, and the verified gotchas. (Strix), worker on Dankserver (RTX 3090), b10433 on both sides, scripts in `scripts/`, morning-of test runbook. Verified gotchas: no `--device` with `--rpc`, default split is proportional to reported free memory (pin 1,1), @@ -57,7 +53,7 @@ graph LR ## Sample Scripts -- [[Sample Script Set]] — a two-node llama.cpp RPC split of +- [[Sample Script Set]] — An educational two-node RPC split script set, for reading only — never run it or deploy it. Qwen3.6-27B-Q8_K_P with MTP, DFlash, and ngram-mod variants, sized for the largest context that fits. For educational reading only. Do not run it in production. @@ -68,3 +64,9 @@ graph LR - [[frank-zealot]] - [[Dankserver System Specs]] - [[Dankclaw System Specs]] + +## Pages + +| Page | Owns | +|---|---| +| [[Multi-Node Runtimes]] | How each two-machine runtime is built and launched, what it can combine with, and the mixed-vendor verdicts. | diff --git a/serverAppConfigs-docs/LLM/Pipeline Parallelism/llama.cpp RPC.md b/serverAppConfigs-docs/LLM/Pipeline Parallelism/llama.cpp RPC.md deleted file mode 100644 index 4b64d9b..0000000 --- a/serverAppConfigs-docs/LLM/Pipeline Parallelism/llama.cpp RPC.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: reference -title: "llama.cpp RPC" -description: "llama.cpp distributed inference via ggml-rpc-server and the --rpc flag: architecture, CLI, versions, limitations, and RPC-compatible speculative decoding." -tags: - - pipeline-parallelism - - research -timestamp: 2026-08-14 ---- diff --git a/serverAppConfigs-docs/LLM/Pipeline Parallelism/vLLM Pipeline Parallelism.md b/serverAppConfigs-docs/LLM/Pipeline Parallelism/vLLM Pipeline Parallelism.md deleted file mode 100644 index 0f4fd32..0000000 --- a/serverAppConfigs-docs/LLM/Pipeline Parallelism/vLLM Pipeline Parallelism.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -type: reference -title: "vLLM Pipeline Parallelism" -description: "vLLM multi-node pipeline parallelism: flags, Ray/mp backends, interconnect requirements, PP bubble, and PP + speculative decoding support." -tags: - - pipeline-parallelism - - research -timestamp: 2026-08-14 ---- diff --git a/serverAppConfigs-docs/LLM/Quantization Production.md b/serverAppConfigs-docs/LLM/Quantization Production.md index dae540b..92af878 100644 --- a/serverAppConfigs-docs/LLM/Quantization Production.md +++ b/serverAppConfigs-docs/LLM/Quantization Production.md @@ -1,3 +1,13 @@ +--- +type: "howto" +title: "Quantization Production" +description: "Making a quantized payload here: the conversion recipe, the scoring gate, and the campaign ladder behind a shipped file." +tags: + - llm + - quantization +timestamp: 2026-10-19 +--- + # Quantization Production — making our own quants, per card and runtime **Research page (2026-10-08).** Question: how do WE produce quantizations diff --git a/serverAppConfigs-docs/LLM/RDNA4 Container Serving.md b/serverAppConfigs-docs/LLM/RDNA4 Container Serving.md index d722203..57cc540 100644 --- a/serverAppConfigs-docs/LLM/RDNA4 Container Serving.md +++ b/serverAppConfigs-docs/LLM/RDNA4 Container Serving.md @@ -1,7 +1,7 @@ --- type: reference title: RDNA4 Container Serving -description: Bringing a model larger than VRAM onto Dankserver's 2x R9700 (gfx1201) with the tcclaviger vLLM stack — quant-class gating, fetch verification, container GPU-init triage, memory budgeting and the expert-offload plan. +description: "Bringing a model larger than VRAM onto the two-card box: quant gating, fetch verification, container GPU-init triage." tags: [rdna4, gfx1201, r9700, vllm, expert-offload, podman, rocm] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/LLM/RDNA4 MoE Expert Offload VRAM Budget.md b/serverAppConfigs-docs/LLM/RDNA4 MoE Expert Offload VRAM Budget.md index d98b8df..3633dd2 100644 --- a/serverAppConfigs-docs/LLM/RDNA4 MoE Expert Offload VRAM Budget.md +++ b/serverAppConfigs-docs/LLM/RDNA4 MoE Expert Offload VRAM Budget.md @@ -1,7 +1,7 @@ --- type: reference title: RDNA4 MoE Expert Offload VRAM Budget -description: The expert-offload plan arithmetic on the tcclaviger fork — which terms a deploy can move, the util law, the CUDA-graph charge, what free VRAM really is, knife-edge survivability, parallelism ceilings and the context ladder. +description: "Expert-offload arithmetic on the RDNA4 fork: movable terms, the utilisation law, graph charge, ceilings, context ladder." tags: [vllm, expert-offload, vram, arena, kv-cache, ladder, tcclaviger] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/LLM/Running Qwen3.6 27B FP8 on Dankclaw.md b/serverAppConfigs-docs/LLM/Running Qwen3.6 27B FP8 on Dankclaw.md index fb00bc0..52c66b0 100644 --- a/serverAppConfigs-docs/LLM/Running Qwen3.6 27B FP8 on Dankclaw.md +++ b/serverAppConfigs-docs/LLM/Running Qwen3.6 27B FP8 on Dankclaw.md @@ -1,6 +1,6 @@ --- type: reference -description: "Qwen3.6-27B is a dense vision-language model from the Qwen3.6 series." +description: "Serving a dense FP8 vision-language checkpoint on the Strix Halo iGPU: the load path, the flags, the format gate." tags: - llm timestamp: 2026-09-03 diff --git a/serverAppConfigs-docs/LLM/Running Qwen3.6 35B A3B FP8 on Dankserver.md b/serverAppConfigs-docs/LLM/Running Qwen3.6 35B A3B FP8 on Dankserver.md index d393e97..8ab4837 100644 --- a/serverAppConfigs-docs/LLM/Running Qwen3.6 35B A3B FP8 on Dankserver.md +++ b/serverAppConfigs-docs/LLM/Running Qwen3.6 35B A3B FP8 on Dankserver.md @@ -1,6 +1,6 @@ --- type: reference -description: "Qwen3.6-35B-A3B is a Mixture-of-Experts vision-language model from the Qwen3.6 series." +description: "Serving a mixture-of-experts FP8 checkpoint on the R9700 host: the load path, the format gate, the offload shape." tags: - llm timestamp: 2026-09-03 diff --git a/serverAppConfigs-docs/LLM/Tcclaviger Fork Ops.md b/serverAppConfigs-docs/LLM/Tcclaviger Fork Ops.md index 6bc2a45..6377ab3 100644 --- a/serverAppConfigs-docs/LLM/Tcclaviger Fork Ops.md +++ b/serverAppConfigs-docs/LLM/Tcclaviger Fork Ops.md @@ -1,7 +1,7 @@ --- type: reference title: Tcclaviger Fork Ops -description: Operating the tcclaviger RDNA4 vLLM fork on dankserver (shape G era, rollback target since the 2026-10-05 FreeToken cutover) — the ad-hoc sweep harness, measured knob verdicts, sweep hygiene, the large-MoE serving recipe, and compile-cache persistence. +description: "Operating the RDNA4 vLLM fork: the sweep harness, measured knob verdicts, sweep hygiene, compile-cache persistence." tags: [tcclaviger, vllm, dankserver, expert-offload, sweeps, cache] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/LLM/VLLM Index.md b/serverAppConfigs-docs/LLM/VLLM Index.md index 69d687a..3275bdb 100644 --- a/serverAppConfigs-docs/LLM/VLLM Index.md +++ b/serverAppConfigs-docs/LLM/VLLM Index.md @@ -1,6 +1,6 @@ --- type: reference -description: "This index contains all documentation for vLLM deployment on Dankserver and Dankclaw." +description: "Locator for every vLLM page in this vault." tags: - llm - vllm diff --git a/serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md b/serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md index f9ffc23..32017c3 100644 --- a/serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md +++ b/serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md @@ -1,7 +1,7 @@ --- -type: research +type: reference title: Dankclaw IOMMU boot line — decision record (amd_iommu=off 2026-09-11 → iommu=pt 2026-10-18) -description: Dankclaw boots iommu=pt since 2026-10-18 — identity DMA with the IOMMU armed, which restored the XDNA2 NPU at a measured prefill cost of nothing (+2.0% on byte-identical traffic). Records the 2026-09-11 amd_iommu=off era that bought a vendor-claimed 13-16% figure never measured here, what that cost, the AMD PerfOpt research (Linux 7.4), the tooling that owns the boot line, and the revert path. +description: "Why Dankclaw boots with identity DMA, what the earlier off-era bought and cost, and which tooling owns the boot line." tags: - dankclaw - iommu diff --git a/serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md b/serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md index 40339e4..31732b4 100644 --- a/serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md +++ b/serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md @@ -1,7 +1,7 @@ --- -type: research +type: reference title: Frontier MoE on Dankserver — what fits, on which runtime -description: VERIFIED 2026-09-28 by measurement and source reading. Compares DeepSeek-V4.1-Flash, MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash against the 2x R9700 (gfx1201) / 246 GiB box, and assesses the Affinity runtime's portability off DeepSeek-V4-Flash. Records the capacity arithmetic that decides the answer before any model is loaded. +description: "The capacity arithmetic that decides whether a frontier mixture-of-experts fits this box before any model is loaded." tags: - dankserver - r9700 diff --git a/serverAppConfigs-docs/LLM/dealign-ornith-crack.md b/serverAppConfigs-docs/LLM/dealign-ornith-crack.md index b3e618a..dc261c8 100644 --- a/serverAppConfigs-docs/LLM/dealign-ornith-crack.md +++ b/serverAppConfigs-docs/LLM/dealign-ornith-crack.md @@ -1,7 +1,7 @@ --- type: reference title: dealign Ornith-1.5-35B-A3B CRACK on Dankserver -description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with ngram-mod speculative decoding; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-zealot. +description: "Retired era: a mixture-of-experts checkpoint served with CPU expert offload and n-gram drafting on a single card." tags: - llamacpp - cuda @@ -9,6 +9,7 @@ tags: - moe - ornith - mtp + - retired-endpoint timestamp: 2026-08-24 --- @@ -123,4 +124,4 @@ llama-server \ - [[deployment]] — deploy and update process - [[Dankserver System Specs]] — the box -- [[frank-redo-2026-08-17]] — fleet topology history \ No newline at end of file +- [[frank-redo-2026-08-17]] — fleet topology histor diff --git a/serverAppConfigs-docs/LLM/deployment.md b/serverAppConfigs-docs/LLM/deployment.md index 93f6d07..8e528b9 100644 --- a/serverAppConfigs-docs/LLM/deployment.md +++ b/serverAppConfigs-docs/LLM/deployment.md @@ -1,5 +1,5 @@ --- -type: guide +type: howto title: LLM Deployment description: Scoped deployment and update process for the three inference services (frank-archon, frank-dragoon, frank-zealot). tags: diff --git a/serverAppConfigs-docs/LLM/frank-archon.md b/serverAppConfigs-docs/LLM/frank-archon.md index 8f01e08..d5856f2 100644 --- a/serverAppConfigs-docs/LLM/frank-archon.md +++ b/serverAppConfigs-docs/LLM/frank-archon.md @@ -1,7 +1,7 @@ --- type: reference title: "frank-archon on Dankserver — radiance (StillDeadcode) 1.0.13, publisher Qwen3.8-Flash-Next iq4r .rad, 2× R9700 TP=2 (ACTIVE)" -description: "The one frank-archon page: the live radiance shape and harness contract (read off scripts/frank-archon.env), the Eras table carrying every rollback path including the re-download doors for the purged payloads, the laws this endpoint taught (pinned-RAM/TTM preflight, the KV dtype law, the one per-card VRAM budget, descriptor selection), the wire contract, and the files that own it. It replaces the nine per-era runbooks." +description: "The radiance endpoint: live shape and harness contract, every era with its rollback path, and the laws it taught." tags: - llm - frank-archon @@ -11,6 +11,7 @@ tags: - r9700 - tp2 - expert-offload + - endpoint timestamp: 2026-10-18 --- @@ -139,8 +140,10 @@ marked "re-fetch" is gone from `/home/dank/models/`, which today holds only `Sti | 2026-09-26 → 09-27 | unsloth Q8_0, 175.30 GiB / 6 parts + shared-Q8_0 MTP 2,786,568,256 B + `mmproj-BF16.gguf` 907,542,944 B | unsloth llama.cpp b11030-mix prebuilt (commit `6ba30d05b`), ROCm gfx120X | `-ts 1,1` layer split, routed experts **and** the PLE table computed in host RAM (`-ot exps=CPU,per_layer_token_embd.weight=CPU`), 262,144 × 1 slot, then the same-day reshape to 4 × 262,144 with V `q8_0` and hot expert blocks 5/3 | the R9V radiance-vLLM fork took the port at Frank's direction (true TP=2 plus MTP4) | trio at commit `4992e5a`+; the GGUF/MTP/mmproj set is no longer on disk → re-fetch, then redeploy | | 2026-09-26 (same day) | `Lucebox/DeepSeek-V4.1-Flash-ROCMFP23-GGUF`, 377.2 GiB, sha256 `2daea6ca…` | `localhost/lucebox-rocm:gfx1201`, built from `Dockerfile.rocm` + 3 local patches | ONE R9700, `--max-ctx 131072`, `--chunk 8192`, q4_0 K+V, `--ds4-prefill sparse`, routed experts on CPU from the page cache | 377 GiB fits neither 63.7 GiB of VRAM nor 246 GiB of RAM → NVMe page-cache thrash; Qwen Flash-Next Q8 gave better quality, vision and 262k at comparable speed | image still in dank's rootless storage; payload gone; the quadlet and its entrypoint were deleted by `8500600` — `git show 8500600^:dankserver/frank-primary/quadlet/frank-primary.container` (and `…:dankserver/frank-primary/frank-primary-entrypoint.sh`), stop the root unit, start the user quadlet | -Before 2026-09-26 the port belonged to the **`frank-medium`** service: [[frank-medium-radiance-mxfp4]] -(vllm-radiance TP=2 MXFP4, 2026-09-20 → 09-26) is that era's page and is **not** superseded here. +Before 2026-09-26 the port belonged to the **`frank-medium`** service, and its era +pages are **not** superseded here: [[frank-medium]] is the locator for that +endpoint's whole shape history, including the unresolved end date of the +vllm-radiance era (this table opens on 09-26; the era page claims 10-10). ## Laws learned here @@ -438,3 +441,4 @@ Related: [[frank-dragoon]] (the abliterated family), [[frank-zealot]], [[Dual-R9700-Cooperation]], [[deployment]], [[swift15-campaign-worklog]], [[dankserver-frontier-moe-feasibility]], [[x4-pair-experiment]], [[RDNA4 Container Serving]], [[frank-medium-radiance-mxfp4]], [[LLM/index]]. +--- diff --git a/serverAppConfigs-docs/LLM/frank-dragoon.md b/serverAppConfigs-docs/LLM/frank-dragoon.md index e06fa33..1312bac 100644 --- a/serverAppConfigs-docs/LLM/frank-dragoon.md +++ b/serverAppConfigs-docs/LLM/frank-dragoon.md @@ -1,7 +1,7 @@ --- type: reference title: "frank-dragoon on Dankclaw — halogen-flash-server 0.16.2, native v2 abliterated Qwen3.8-Flash-Next (canonical)" -description: "The one runbook for frank-dragoon, the Strix Halo iGPU endpoint (192.168.0.143:7770): current shape from the env, every retired era with its rollback path, the durable laws this endpoint paid for, the wire contract, and the files that own it." +description: "The iGPU endpoint: live shape, every era with its rollback path, the laws it paid for, and the files that own it." tags: - llm - frank-dragoon @@ -9,6 +9,7 @@ tags: - strix-halo - gfx1151 - halogen + - endpoint timestamp: 2026-10-18 --- @@ -234,3 +235,4 @@ created it — earlier topologies in [[frank-redo-2026-08-17]]. - Related: [[deployment]] · [[Model Swap and Rollback]] · [[frank-zealot]] · [[frank-archon]] · [[dankclaw-iommu-translation-off]] · [[frank-redo-2026-08-17]] · [[Dankclaw System Specs]] · [[DFlash DSpark Speculative Decoding]] · [[qwen3.8-spec-matrix]] · [[qwen3.8-strix-optimizations]]. +--- diff --git a/serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md b/serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md index 63b726e..706ed55 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md +++ b/serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md @@ -2,7 +2,7 @@ timestamp: 2026-09-17 type: reference title: frank-medium — Batch and Ubatch Sweep -description: llama-bench sweep of --batch-size against --ubatch-size on frank-medium (2x Radeon AI PRO R9700, layer split) at the deployed context shape, and the rebalance it forced. Prefill peaks at ubatch 512 and falls on BOTH sides. +description: "The batch-versus-ubatch sweep on the layer-split pair at its deployed shape, and the rebalance the sweep forced." tags: - benchmark - llamacpp @@ -11,6 +11,7 @@ tags: - dankserver - layer-split --- + - retired-endpoint # frank-medium — Batch and Ubatch Sweep @@ -222,4 +223,5 @@ re-checking that 35:30 and 37:28 still fail so the gate keeps discriminating. difference: with no KV to attend over, per-pass overhead dominates and a bigger micro-batch wins; at 64k of resident KV it does not. - [[Dual-R9700-Cooperation]] — why the cards are layer-split at all -- [[Dankserver System Specs]] — hardware inventory +- [[Dankserver System Specs]] — hardware inventory--- +timestamp: 2026-09-17 diff --git a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md index 7909ff5..459ad6e 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md +++ b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md @@ -1,7 +1,7 @@ --- -type: configuration +type: reference title: frank-medium llama.cpp Runbook (DFlash2 drafter) -description: The former ACTIVE frank-medium deployment since 2026-09-16 (model swapped 2026-09-19) — mainline llama.cpp on Dankserver serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the z-lab DFlash2 Q4_K_M drafter (the model's own MTP head stays in the file, one SPEC_TYPE change away), as ONE layer-split process across both Radeon AI PRO R9700 cards at a balanced 32:33 split, a 327680-position KV pool (163840 per conversation over two slots), f16 KV plus a 16 GiB host-RAM prompt cache, vision via the BF16 mmproj. +description: "Retired era: the layer-split llama.cpp shape with a drafter, its ladder measurements, and the balance it settled on." tags: - frank-medium - dankserver @@ -12,6 +12,7 @@ tags: - gfx1201 - r9700 - layer-split + - retired-endpoint timestamp: 2026-09-20 --- @@ -730,3 +731,4 @@ fleet. - [[frank-medium-batch-ubatch]] — the batch/ubatch sweep that set the deployed geometry and moved this split from 36:29 to 35:30: the ubatch optimum at 512, why the logical batch is free, and the harness traps +--- diff --git a/serverAppConfigs-docs/LLM/frank-medium-model-swap.md b/serverAppConfigs-docs/LLM/frank-medium-model-swap.md index 1ab4642..4499c5c 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-model-swap.md +++ b/serverAppConfigs-docs/LLM/frank-medium-model-swap.md @@ -1,7 +1,7 @@ --- -type: runbook +type: howto title: frank-medium Model Swap -description: RETIRED-era procedure (frank-medium is a retired endpoint; `:7770` on Dankserver is [[frank-archon]] today and the live swap law is [[Model Swap and Rollback]]) — how a checkpoint swap behind frank-medium worked: the streamlining toolchain (hf-manifest.py, hf-fetch.sh, swap-frank-medium.sh), the quantization-aware entrypoint contract, the quadlet edit, the deploy, the capability battery, and the rollback path. +description: "Retired-era swap procedure: the fetch toolchain, the entrypoint contract, the capability battery, the rollback path." tags: - frank-medium - dankserver @@ -9,6 +9,7 @@ tags: - vllm-radiance - huggingface - runbook + - retired-endpoint timestamp: 2026-09-20 --- @@ -155,9 +156,10 @@ back: 2. `deploy.sh dankserver --service frank-medium --skip-e2e`. 3. Re-run the capability battery. -The 2026-09-20 rollback target is the MXFP4 checkpoint at -`/home/dank/models/Qwen3.8-27B-Uncensored-MXFP4-awq` (19.37 GiB single -safetensors) with `SPEC_METHOD=dflash`, `SPEC=7`. +The 2026-09-20 rollback target is the MXFP4 checkpoint with +`SPEC_METHOD=dflash`, `SPEC=7`; [[frank-medium-radiance-mxfp4]] owns that +payload's identity, and as of the 2026-10-19 host census it is a re-fetch, not a +local file. ## 2026-09-20 swap record (MXFP4 → FP8) @@ -184,3 +186,4 @@ safetensors) with `SPEC_METHOD=dflash`, `SPEC=7`. DFlash2 drafter's 2.1 GiB costs the pool vs MTP's 461,540 (1.76×); both smaller than MXFP4's 943,581 (3.59×) because FP8 weights are bigger (30.9 vs 19.4 GiB). +--- diff --git a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md index d1b99a4..54d27d9 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md +++ b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md @@ -1,7 +1,7 @@ --- -type: configuration +type: reference title: frank-medium vllm-radiance MXFP6 Runbook -description: RETIRED era runbook (endpoint renamed frank-archon 2026-10-10; this shape is a row in [[frank-archon]]'s Eras table) — vllm-radiance serving hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 (ParoQuant MXFP6 W4A8, DFlash2 drafter, unquantized bf16 KV) as ONE tensor-parallel process (TP=2) on Dankserver's two AMD Radeon AI PRO R9700 (gfx1201). Cut to MXFP6 on 2026-09-21 (PR #51's launch6_p kernel, image 0.13.0-e060895); the MXFP4 checkpoint (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq, the 2026-09-21 shape) is the on-disk rollback. Model-swap process: [[frank-medium-model-swap]]. The sections below describe the 2026-09-16 two-replica era. +description: "Retired era: the tensor-parallel radiance shape on two cards, its quant-class swap, and the on-disk rollback payload." tags: - frank-medium - dankserver @@ -14,6 +14,7 @@ tags: - rccl - pcie-atomics - quadlet + - retired-endpoint timestamp: 2026-09-21 --- @@ -37,11 +38,14 @@ e4m3fn scale) with `--kv-cache-dtype auto` (unquantized bf16 KV) and `--gpu-memory-utilization 0.98`. The Swift variant is a finetune of `ukisai/Swift-Qwen3.8-27b` (a different base from `Qwen/Qwen3.8-27B`), so the DFlash2-FP8 drafter's hidden-state distribution is a mismatch — acceptance may -drop from the MXFP4 era's 4.24/7, but it is still a net win over no spec -decode. **The MXFP4 checkpoint is the on-disk rollback**: flip `SNAP` back to -`/models/Qwen3.8-27B-Uncensored-MXFP4-awq`, and the image -back to `localhost/radiance-vllm:0.9.3` (re-pull `docker.io/stilldeadcode/ -vllm-radiance:0.9.3` if it is no longer on disk; the MXFP4 shape auto-sized to +drop from the 4.24/7 that the DFlash2 arm measured on the FP8 target (the A/B +below ran both arms on FP8, not on MXFP4). **The MXFP4 checkpoint is the +rollback**, and as of the 2026-10-19 host census it is **no longer on disk**: +`/home/dank/models/` holds `StillDeadcode`, `rad` and `tpl.jinja` only, so a +rollback re-fetches `/models/Qwen3.8-27B-Uncensored-MXFP4-awq` (Hub +`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq`) and flips `SNAP` and the +image back to `localhost/radiance-vllm:0.9.3` (re-pull +`docker.io/stilldeadcode/vllm-radiance:0.9.3` if the local tag is gone; the MXFP4 shape auto-sized to 486,464 tokens with bf16 KV and held the full 262,144 window). **Deployed shape since 2026-09-20: one rootless Quadlet, `--tensor-parallel-size 2`.** @@ -528,9 +532,8 @@ soak (three concurrent loops, each re-sending a ~30k-token document every turn with growing history) shows what happens at that limit: turns alternate between **~1.1 s** when the prefix is resident and **12-22 s** when it has been evicted and must be prefilled again, because two families per card plus their cached -prefixes exceed the pool. Nothing failed — ~120 requests, no errors, no failover, -both replicas steady — and per-family pinning kept the fast turns on the replica -that holds the prefix. +prefixes exceed the pool. Nothing failed, and per-family pinning kept the fast +turns on the replica that holds the prefix. Read that as a capacity budget, not a defect: two concurrent large-context sessions per card are comfortable, three or more of that size trade turns @@ -632,3 +635,4 @@ if that person is looking. That is the honest state of "alerting" here. the same image and kernel 7.0.0-30 failing identically. - `ggz14/radiance-vllm-mxfp4` — the patch set and the reference benchmark table this page compares against. +--- diff --git a/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md b/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md index 53a2c4e..b370466 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md +++ b/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md @@ -1,7 +1,7 @@ --- -type: configuration +type: reference title: frank-medium ROCmFPX Runbook -description: charlie12345/ROCmFPX llama.cpp serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF as frank-medium on Dankserver (2x AMD Radeon AI PRO R9700, gfx1201). +description: "Retired era: the ROCmFPX fork serving an agent-tuned checkpoint on the two-card box, and the loader pin it required." tags: - frank-medium - dankserver @@ -10,6 +10,7 @@ tags: - mtp - gfx1201 - r9700 + - retired-endpoint timestamp: 2026-09-15 --- diff --git a/serverAppConfigs-docs/LLM/frank-medium.md b/serverAppConfigs-docs/LLM/frank-medium.md new file mode 100644 index 0000000..98a906e --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-medium.md @@ -0,0 +1,65 @@ +--- +type: reference +title: frank-medium (retired) — the Dankserver endpoint before frank-archon +description: "Locator for the retired Dankserver LLM endpoint: which era page owns which payload, and where the dates disagree." +timestamp: 2026-10-19 +tags: + - llm + - dankserver + - frank-medium + - retired + - retired-endpoint +--- + +# frank-medium (retired) — the Dankserver endpoint before frank-archon + +`frank-medium` was the LLM service on Dankserver port 7770. It is **not** a live +endpoint: the same port is [[frank-archon]] today, and [[frank-archon]] owns every +shape served from 2026-09-26 onward, including the `frank-medium` → `frank-archon` +name cutover dated 2026-10-10 in `AGENTS.md`. This page owns only the **locator** — +which era page holds which payload's evidence — and carries no measurements: every +number stays in the era page that measured it. + +## Eras + +Windows are quoted from each era page's own banner; no date here is derived. + +| Window | Runtime | Payload | Shape | Page | Rollback | +|---|---|---|---|---|---| +| 2026-09-15 → 2026-09-16 | llama.cpp, ROCmFPX fork pinned at commit `36e9acd` | `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` (custom tensor type 103) | single card; the fork is the only loader | [[frank-medium-rocmfpx]] | re-fetch the GGUF, rebuild the fork | +| 2026-09-16 → 2026-09-19 | vllm-radiance on the old TUF GAMING B650-E board | ParoQuant MXFP4 | two single-GPU replicas behind an adaptive-routing proxy | [[frank-medium-radiance-mxfp4]] (sections below its deployed-shape header) | the replica pair is reconstructable from that page's flags | +| 2026-09-16 → 2026-09-20 | llama.cpp `fb27a525`, ROCm `7.14.0a` | its own GGUF build, DFlash drafter | `-sm layer` split across the two cards, drafter pinned to one device | [[frank-medium-llamacpp-dflash2]] | the page is the named rollback target for the llama shape | +| 2026-09-20 → disputed, see Open questions | vllm-radiance on the replacement ProArt board | ParoQuant MXFP4, then MXFP6 (`paroquant_mxfp6`) with bf16 KV | `--tensor-parallel-size 2` over x8/x8 Gen5 root ports | [[frank-medium-radiance-mxfp4]] (deployed-shape header) | re-fetch the MXFP4 checkpoint (host census 2026-10-19: not local); image `localhost/radiance-vllm:0.9.3` | + +`[[frank-medium-batch-ubatch]]` is not an era: it is the batch-versus-ubatch sweep run +on the layer-split pair at its deployed shape, and its numbers belong to that +campaign. `[[frank-medium-model-swap]]` is likewise a campaign ledger — the record of +the 2026-09-20 swap procedure — while the procedure itself is owned by +[[Model Swap and Rollback]]. + +## Laws + +None restated. Each era page carries its own laws and links them to the technique +page that owns the mechanism ([[PCIe Link and Atomics]], +[[Llama Cpp Multi-Device Splits]], [[DFlash DSpark Speculative Decoding]], +[[Llama Cpp Serving Sizing]], [[Dual-R9700-Cooperation]]). + +## Open questions + +Two date claims in the era pages do not reconcile, and both are left standing rather +than silently corrected: + +- `frank-medium-rocmfpx.md` banner says the MXFP4 stack replaced that era and, in the + next sentence, names [[frank-medium-llamacpp-dflash2]] as the active successor. The + MXFP4 replica era and the llama layer-split era both open on 2026-09-16, so which + one directly followed the ROCmFPX fork cannot be settled from the repo. +- `frank-medium-radiance-mxfp4.md` dates its own retired era to 2026-10-10, while + [[frank-archon]]'s Eras table opens on 2026-09-26. The gap is either an unrecorded + fortnight on that endpoint or a stale banner; nothing else in the repo covers + 2026-09-26 → 10-10 for port 7770. + +## Files + +No repo files own this endpoint: its units, env and serve scripts were renamed to +`frank-archon` on 2026-10-10 and are owned by [[frank-archon]]. `scripts/old/` holds +retired serve scripts from these eras as reference only. diff --git a/serverAppConfigs-docs/LLM/frank-npu.md b/serverAppConfigs-docs/LLM/frank-npu.md index cfdce4a..fd7432b 100644 --- a/serverAppConfigs-docs/LLM/frank-npu.md +++ b/serverAppConfigs-docs/LLM/frank-npu.md @@ -1,7 +1,7 @@ --- type: reference title: frank-npu (retired) — FLM NPU endpoint on Dankclaw -description: The retired FastFlowLM endpoint on Dankclaw's XDNA2 NPU — what it served, why it died on 2026-09-11 with the amd_iommu=off decision, and where every dependent route went. +description: "The retired NPU endpoint: what it served, why it died with the IOMMU decision, and where every dependent route went." tags: - flm - fastflowlm @@ -10,6 +10,7 @@ tags: - dankclaw - frank-npu - retired + - retired-endpoint timestamp: 2026-09-11 --- diff --git a/serverAppConfigs-docs/LLM/frank-redo-2026-08-17.md b/serverAppConfigs-docs/LLM/frank-redo-2026-08-17.md index 687be05..0832dca 100644 --- a/serverAppConfigs-docs/LLM/frank-redo-2026-08-17.md +++ b/serverAppConfigs-docs/LLM/frank-redo-2026-08-17.md @@ -1,9 +1,10 @@ --- type: reference -description: "This page records the full redo of the inference fleet on 2026-08-17." +description: "The day the whole inference fleet was rebuilt: what came down, what replaced it, and why it was done that way." tags: - llm - frank + - retired-endpoint timestamp: 2026-09-03 --- @@ -223,3 +224,4 @@ repeated prompt prefixes. Measured on a 2023-token prefix (2026-08-17 evening): 4.6s cold -> 1.1s cached (~4x TTFT), output unchanged. This is the main TTFT lever for pi, which re-sends the same system prompt + tool schema every turn. Toggled via `ENABLE_PREFIX_CACHING` in the env file. +--- diff --git a/serverAppConfigs-docs/LLM/frank-zealot.md b/serverAppConfigs-docs/LLM/frank-zealot.md index 2b87766..1dc103c 100644 --- a/serverAppConfigs-docs/LLM/frank-zealot.md +++ b/serverAppConfigs-docs/LLM/frank-zealot.md @@ -1,7 +1,7 @@ --- type: reference title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Ternary-Bonsai-2-9B PQ2_0-MTP, 2 × 262144, exact f16 KV" -description: "The one frank-zealot reference: the live shape on Dankclaw's RTX 3090 (:7771) exactly as scripts/frank-zealot.env ships it, every era with its rollback path, the deduplicated laws of this card, the wire contract, and the owning files." +description: "The RTX 3090 endpoint: live shape exactly as the env ships it, every era with its rollback path, and its card laws." tags: - llm - frank-zealot @@ -9,6 +9,7 @@ tags: - llama-cpp - gguf - rtx3090 + - endpoint timestamp: 2026-10-18 --- diff --git a/serverAppConfigs-docs/LLM/freetoken-benchmark-sweep.md b/serverAppConfigs-docs/LLM/freetoken-benchmark-sweep.md index 2ab0249..df6eaa4 100644 --- a/serverAppConfigs-docs/LLM/freetoken-benchmark-sweep.md +++ b/serverAppConfigs-docs/LLM/freetoken-benchmark-sweep.md @@ -1,7 +1,7 @@ --- -type: research +type: reference title: FreeToken Benchmark Sweep — Dankserver RTX 3090 -description: KV pool vs decode speed sweep across bs1/bs3/bs4/bs6 on Dankserver with Qwen3.6-35B-A3B-FP8 via FreeToken v0.1.2. +description: "KV pool versus decode speed for the FP8 mixture-of-experts payload on the FreeToken runtime." tags: - freetoken - benchmark diff --git a/serverAppConfigs-docs/LLM/ikllama-ornith-dankserver.md b/serverAppConfigs-docs/LLM/ikllama-ornith-dankserver.md index e6abe58..3647ce2 100644 --- a/serverAppConfigs-docs/LLM/ikllama-ornith-dankserver.md +++ b/serverAppConfigs-docs/LLM/ikllama-ornith-dankserver.md @@ -1,7 +1,7 @@ --- type: reference title: ik_llama.cpp on Dankserver (Ornith Q8_0) -description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-zealot on the RTX 3090. Q8_0 model, KV K default f16 / V q6_0 with Hadamard on V only (iq5_k is not a KV-cache type in this build; q8_KV crashes on this arch), split-mode none, speculative decoding disabled, 3 parallel slots x 163840 ctx, the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags. +description: "Retired era: the ik_llama fork with a quantized value cache and CPU expert offload, and the build flags it required." tags: - ikllama - llamacpp @@ -11,7 +11,8 @@ tags: - ornith - kv-cache - hadamard -timestamp: 2026-08-26 + - retired-endpoint +timestamp: 2026-10-19 --- # ik_llama.cpp on Dankserver (Ornith-1.5-35B-A3B CRACK) @@ -259,4 +260,4 @@ spec / `--spec-type` flags. The model file is not changed either way. build flags incl. `GGML_CUDA_FA_ALL_QUANTS` and `GGML_IQK_FA_ALL_QUANTS` - [ik_llama.cpp serverAppConfigs-docs/parameters.md](https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/parameters.md) — q8_KV (PR 208), K/V Hadamard options, `-cuda graphs=0` workaround, and - `--split-mode graph` MoE note (PR 1080) \ No newline at end of file + `--split-mode graph` MoE note (PR 1080) diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index d8367ef..0f4ecf4 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -9,37 +9,33 @@ each of those owns exactly that one thing. The incidents behind the laws are in | Page | Owns | |---|---| -| [[frank-archon]] | The one frank-archon page: the live radiance shape and harness contract (read off scripts/frank-archon.env), the Eras… | -| [[frank-dragoon]] | The one runbook for frank-dragoon, the Strix Halo iGPU endpoint (192.168.0.143:7770): current shape from the env,… | -| [[frank-zealot]] | The one frank-zealot reference: the live shape on Dankclaw's RTX 3090 (:7771) exactly as scripts/frank-zealot.env… | +| [[frank-archon]] | The radiance endpoint: live shape and harness contract, every era with its rollback path, and the laws it taught. | +| [[frank-dragoon]] | The iGPU endpoint: live shape, every era with its rollback path, the laws it paid for, and the files that own it. | +| [[frank-zealot]] | The RTX 3090 endpoint: live shape exactly as the env ships it, every era with its rollback path, and its card laws. | ## Retired endpoints | Page | Owns | |---|---| -| [[frank-medium-batch-ubatch]] | llama-bench sweep of --batch-size against --ubatch-size on frank-medium (2x Radeon AI PRO R9700, layer split) at the… | -| [[frank-medium-llamacpp-dflash2]] | The former ACTIVE frank-medium deployment since 2026-09-16 (model swapped 2026-09-19) — mainline llama.cpp on… | -| [[frank-medium-model-swap]] | RETIRED-era swap procedure for the frank-medium endpoint (`:7770` is now [[frank-archon]]) — the… | -| [[frank-medium-radiance-mxfp4]] | RETIRED frank-medium era runbook — vllm-radiance serving hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 (ParoQuant MXFP6 W4A8,… | -| [[frank-medium-rocmfpx]] | charlie12345/ROCmFPX llama.cpp serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF as frank-medium on Dankserver… | -| [[frank-npu]] | The retired FastFlowLM endpoint on Dankclaw's XDNA2 NPU — what it served, why it died on 2026-09-11 with the… | -| [[frank-redo-2026-08-17]] | This page records the full redo of the inference fleet on 2026-08-17. | -| [[ikllama-ornith-dankserver]] | ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-zealot on the RTX 3090. Q8_0 model, KV K default f16 / V… | -| [[kingjones-q8agent-rocmfpx]] | **Status (historical, 2026-09-13):** was to deploy on Dankclaw. | -| [[qwen3.8-spec-matrix]] | This page records the speculative-decoding benchmark history and the | +| [[frank-medium-batch-ubatch]] | The batch-versus-ubatch sweep on the layer-split pair at its deployed shape, and the rebalance the sweep forced. | +| [[frank-medium-llamacpp-dflash2]] | Retired era: the layer-split llama.cpp shape with a drafter, its ladder measurements, and the balance it settled on. | +| [[frank-medium-model-swap]] | Retired-era swap procedure: the fetch toolchain, the entrypoint contract, the capability battery, the rollback path. | +| [[frank-medium-radiance-mxfp4]] | Retired era: the tensor-parallel radiance shape on two cards, its quant-class swap, and the on-disk rollback payload. | +| [[frank-medium-rocmfpx]] | Retired era: the ROCmFPX fork serving an agent-tuned checkpoint on the two-card box, and the loader pin it required. | +| [[frank-npu]] | The retired NPU endpoint: what it served, why it died with the IOMMU decision, and where every dependent route went. | +| [[frank-redo-2026-08-17]] | The day the whole inference fleet was rebuilt: what came down, what replaced it, and why it was done that way. | +| [[ikllama-ornith-dankserver]] | Retired era: the ik_llama fork with a quantized value cache and CPU expert offload, and the build flags it required. | +| [[kingjones-q8agent-rocmfpx]] | Historical: a checkpoint and loader pairing staged for this host and never deployed on it. | +| [[qwen3.8-spec-matrix]] | The speculative-decoding benchmark history for this endpoint, arm by arm, and what each arm ruled out. | ## Pipeline Parallelism (series) | Page | Owns | |---|---| -| [[Ethernet Bandwidth Math]] | Bytes-per-token math for pipeline vs tensor parallelism, feasibility tables for 1/2.5/10 GbE, and real multi-node… | -| [[Expected Performance]] | Estimated tokens per second for a 25-40B model split across two nodes over 1/2.5/10 GbE, with real benchmark anchors. | -| [[llama.cpp RPC]] | llama.cpp distributed inference via ggml-rpc-server and the --rpc flag: architecture, CLI, versions, limitations, and… | -| [[Mixed-Vendor Experiments]] | Real-world NVIDIA + AMD Strix Halo multi-node experiments: what works (llama.cpp RPC, Vulkan split), what breaks… | -| [[Qwen3.6 27B Spec Methods]] | Qwen3.6-27B Q8 GGUF variants and llama.cpp speculative decoding (draft-mtp, draft-dflash, ngram-mod): flags, RPC… | -| [[Qwen3.8 PP Deployment]] | Split Qwen3.8-27B across the Strix Halo iGPU (head) and the RTX | -| [[Sample Script Set]] | An educational two-node llama.cpp RPC split of Qwen3.6-27B-Q8_K_P over ethernet. For reading only. Do not run or… | -| [[vLLM Pipeline Parallelism]] | vLLM multi-node pipeline parallelism: flags, Ray/mp backends, interconnect requirements, PP bubble, and PP +… | +| [[Ethernet Bandwidth Math]] | Bytes-per-token math for pipeline versus tensor parallelism, and the link budgets that follow from it. | +| [[Expected Performance]] | Estimated split-model throughput over ethernet, with the benchmark anchors the estimates rest on. | +| [[Qwen3.8 PP Deployment]] | The real split deployment: which node is head and which is worker, the flags, and the verified gotchas. | +| [[Sample Script Set]] | An educational two-node RPC split script set, for reading only — never run it or deploy it. | ## Techniques, hardware, trials @@ -47,43 +43,48 @@ each of those owns exactly that one thing. The incidents behind the laws are in |---|---| | [[Dankclaw Improvements]] | This page collects concrete, sourced recommendations to improve Dankclaw | | [[Dankclaw System Specs]] | Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. | -| [[dankclaw-iommu-translation-off]] | Dankclaw boots iommu=pt since 2026-10-18 — identity DMA with the IOMMU armed, which restored the XDNA2 NPU at a… | -| [[Dankserver System Specs]] | Dankserver hardware and OS state since the 2026-09-20 board swap (ASUS ProArt B850-Creator WiFi NEO, 2x AMD Radeon AI… | -| [[dankserver-frontier-moe-feasibility]] | VERIFIED 2026-09-28 by measurement and source reading. Compares DeepSeek-V4.1-Flash, MiMo-V2.6-Flash-MOPD and… | +| [[dankclaw-iommu-translation-off]] | Why Dankclaw boots with identity DMA, what the earlier off-era bought and cost, and which tooling owns the boot line. | +| [[Dankserver System Specs]] | Dankserver hardware and OS state since the board swap: board, lane topology, cards, memory, kernel posture. | +| [[dankserver-frontier-moe-feasibility]] | The capacity arithmetic that decides whether a frontier mixture-of-experts fits this box before any model is loaded. | | [[dankserver-vram-growth]] | Investigation of the recurring cuBLAS crash on frank-zealot. What grows, what does not, and what to do next. | -| [[dealign-ornith-crack]] | dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with… | +| [[dealign-ornith-crack]] | Retired era: a mixture-of-experts checkpoint served with CPU expert offload and n-gram drafting on a single card. | | [[deployment]] | Scoped deployment and update process for the three inference services (frank-archon, frank-dragoon, frank-zealot). | | [[DFlash DSpark Speculative Decoding]] | DFlash and DSpark are the two newest speculative decoding methods. | -| [[Dual-R9700-Cooperation]] | Why frank-medium's two Radeon AI PRO R9700s could not run tensor-parallel on the old B650-E board (PCIe atomic… | -| [[FastFlowLM NPU Inference]] | FastFlowLM (FLM), the AMD NPU-first LLM runtime for Ryzen AI XDNA2 chips. Fit analysis for the 50 TOPS NPU on… | -| [[FreeToken Builds]] | Building FreeToken (the official franksdank fork of FreeTok Tokenworks' freetoken vLLM-MTP runtime) on Dankclaw — the… | -| [[freetoken-benchmark-sweep]] | KV pool vs decode speed sweep across bs1/bs3/bs4/bs6 on Dankserver with Qwen3.6-35B-A3B-FP8 via FreeToken v0.1.2. | -| [[GGUF Compatibility]] | How to answer "will engine X load this GGUF?" from the file's header alone, why a custom tensor type forces the… | -| [[Heterogeneous Split Triage]] | Triage before building when someone asks to split a model across unequal accelerators on one host (24 GB GDDR6X +… | +| [[Dual-R9700-Cooperation]] | Why two R9700s could not run tensor-parallel on the old board, and what the board swap unblocked. | +| [[FastFlowLM NPU Inference]] | The AMD NPU-first runtime: fit analysis for the XDNA2, published benchmarks, install state, and test runbook. | +| [[FreeToken Builds]] | Building the FreeToken runtime here: the CUDA and ROCm builds and the dependency trap each one hit. | +| [[freetoken-benchmark-sweep]] | KV pool versus decode speed for the FP8 mixture-of-experts payload on the FreeToken runtime. | +| [[GGUF Compatibility]] | Answering whether an engine loads a GGUF from the file header alone, and pinning a fork build that can. | +| [[Heterogeneous Split Triage]] | Triage before splitting across unequal accelerators: link characterisation, real bottleneck, what placement never buys. | | [[Hybrid CPU GPU Processing Dankserver]] | Dankserver uses NVIDIA GPU with system RAM for hybrid processing. | -| [[ling-3.0-flash-trial]] | `bailingmoe3-support` (commit `db4480bc8`, build b10411) at `/home/dank/llama.cpp-ling`. | -| [[ling-3.0-flash-vllm-plan]] | This page was the plan for a vLLM-ROCm deployment of `jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native` (INT4 W4A16, ~62… | -| [[Llama Cpp Multi-Device Splits]] | What llama.cpp can and cannot express across devices, the no-collective layer-split path, split balancing by… | -| [[Llama Cpp Serving Sizing]] | Slot-pool sizing against a headroom band, per-device VRAM accounting, prompt-cache/state sizing and the QSA… | -| [[LLM Bench Windows and Measurement]] | Stopping-and-restoring benchmark windows on the fleet LLM hosts (llama-bench sweeps, HTTP param benches, multi-arm… | -| [[LLM Performance Diagnosis]] | Probing why a served LLM is slow — the flat-depth probe, the expert-offload duty-cycle arithmetic, kernel-time… | -| [[Model Swap and Rollback]] | The full model-swap discipline for the LLM fleet — fit preflight, fetch and payload verification, runtime build/flag… | -| [[PCIe Link and Atomics]] | Link probing (idle downshift, path ceilings, CUDA/HIP bandwidth probes), the streaming-vs-compute ceiling arithmetic,… | +| [[ling-3.0-flash-trial]] | Trial of a fork build for a new mixture-of-experts architecture: what loaded, what the harness refused, and the verdict. | +| [[Llama Cpp Multi-Device Splits]] | What llama.cpp can express across devices: the no-collective layer-split path, balancing, and drafter KV sizing. | +| [[Llama Cpp Serving Sizing]] | Measurement-first sizing for llama.cpp shapes: slot pools against a headroom band, per-device accounting, prompt cache. | +| [[LLM Bench Windows and Measurement]] | Opening and restoring benchmark windows on fleet hosts, honest methodology, and the usage-counter law under drafting. | +| [[LLM Performance Diagnosis]] | Probing why a served model is slow: the flat-depth probe, offload duty-cycle arithmetic, and profiler attribution. | +| [[Model Swap and Rollback]] | The model-swap discipline: fit preflight, fetch verification, build and flag checks, live validation, three-pin revert. | +| [[PCIe Link and Atomics]] | Link probing, the streaming-versus-compute ceiling arithmetic, and the atomics wall that decides multi-GPU collectives. | | [[Quantization by Hardware Architecture]] | This page maps LLM quantization formats to the GPU and CPU architectures that support them. | -| [[Quantization Production]] | — | -| [[qwen3.8-strix-optimizations]] | This page answers one question: **how do we make Qwen3.8-27B serve faster on | +| [[Quantization Production]] | Making a quantized payload here: the conversion recipe, the scoring gate, and the campaign ladder behind a shipped file. | +| [[qwen3.8-strix-optimizations]] | How a served checkpoint got faster on the Strix Halo iGPU: the lever ladder and what each lever actually bought. | | [[RAM CPU Offloading in VLLM]] | vLLM uses CPU RAM and swap space to extend GPU memory capacity. | -| [[RDNA4 Container Serving]] | Bringing a model larger than VRAM onto Dankserver's 2x R9700 (gfx1201) with the tcclaviger vLLM stack — quant-class… | -| [[RDNA4 MoE Expert Offload VRAM Budget]] | The expert-offload plan arithmetic on the tcclaviger fork — which terms a deploy can move, the util law, the… | -| [[Running Qwen3.6 27B FP8 on Dankclaw]] | Qwen3.6-27B is a dense vision-language model from the Qwen3.6 series. | -| [[Running Qwen3.6 35B A3B FP8 on Dankserver]] | Qwen3.6-35B-A3B is a Mixture-of-Experts vision-language model from the Qwen3.6 series. | +| [[RDNA4 Container Serving]] | Bringing a model larger than VRAM onto the two-card box: quant gating, fetch verification, container GPU-init triage. | +| [[RDNA4 MoE Expert Offload VRAM Budget]] | Expert-offload arithmetic on the RDNA4 fork: movable terms, the utilisation law, graph charge, ceilings, context ladder. | +| [[Running Qwen3.6 27B FP8 on Dankclaw]] | Serving a dense FP8 vision-language checkpoint on the Strix Halo iGPU: the load path, the flags, the format gate. | +| [[Running Qwen3.6 35B A3B FP8 on Dankserver]] | Serving a mixture-of-experts FP8 checkpoint on the R9700 host: the load path, the format gate, the offload shape. | | [[Shared Memory Architecture Dankclaw]] | Dankclaw uses AMD Strix Halo APU with unified memory architecture. | -| [[swift15-campaign-worklog]] | — | -| [[Tcclaviger Fork Ops]] | Operating the tcclaviger RDNA4 vLLM fork on dankserver (shape G era, rollback target since the 2026-10-05 FreeToken… | -| [[ubatch-benchmark-large]] | Standardized ubatch sweep for the active kingjones Qwen3.8 ROCmFPX deployment. | -| [[vLLM Boot Forensics and Config Interrogation]] | Capturing a vLLM container boot failure's real root cause from journald (the container log is already gone),… | -| [[VLLM Index]] | This index contains all documentation for vLLM deployment on Dankserver and Dankclaw. | +| [[swift15-campaign-worklog]] | Dated worklog of the campaign that converted and served a Swift checkpoint on the radiance endpoint, arm by arm. | +| [[Tcclaviger Fork Ops]] | Operating the RDNA4 vLLM fork: the sweep harness, measured knob verdicts, sweep hygiene, compile-cache persistence. | +| [[ubatch-benchmark-large]] | The wide ubatch sweep for a deployed ROCmFPX shape, run through the standardized harness. | +| [[vLLM Boot Forensics and Config Interrogation]] | Recovering a container boot failure's real cause from journald, and interrogating a live endpoint's true config. | +| [[VLLM Index]] | Locator for every vLLM page in this vault. | | [[VLLM Overview]] | vLLM is a fast and easy library for large language model inference and serving. | -| [[vLLM Porting and Compile Triage]] | Forward-porting a model architecture to a new vLLM major (additive-only discipline, gap enumeration, shims that… | -| [[vllm_research_findings]] | vLLM v0.26.0 (latest stable, released July 25, 2026) supports DFlash speculative decoding with the `--spec-method… | -| [[x4-pair-experiment]] | What mainline llama.cpp can and cannot do with a 3090 + Strix Halo combination, measured 2026-09-16 on Dankclaw — the… | +| [[vLLM Porting and Compile Triage]] | Forward-porting a model architecture: additive-only discipline, shims that survive worker spawn, compile triage. | +| [[vllm_research_findings]] | Upstream support survey for speculative decoding: which flags, releases and pull requests carry each method. | +| [[x4-pair-experiment]] | Measured limits of a consumer card plus APU pair: the tensor inventory, the flag that kills cross-device placement. | + +## Pages + +| Page | Owns | +|---|---| +| [[frank-medium]] | Locator for the retired Dankserver LLM endpoint: which era page owns which payload, and where the dates disagree. | diff --git a/serverAppConfigs-docs/LLM/kingjones-q8agent-rocmfpx.md b/serverAppConfigs-docs/LLM/kingjones-q8agent-rocmfpx.md index 729375c..628c91e 100644 --- a/serverAppConfigs-docs/LLM/kingjones-q8agent-rocmfpx.md +++ b/serverAppConfigs-docs/LLM/kingjones-q8agent-rocmfpx.md @@ -1,10 +1,11 @@ --- type: reference -description: "**Status (historical, 2026-09-13):** was to deploy on Dankclaw." +description: "Historical: a checkpoint and loader pairing staged for this host and never deployed on it." tags: - llm - rocm - frank + - retired-endpoint timestamp: 2026-09-15 --- @@ -239,3 +240,4 @@ reads `--host/--port/--temp` as CLI flags, never as env vars. `gl=45`. See the `set_rows` build bug section above. - Preceding model: Kairic Edge Qwen3.8-27B IU4 (deployed 2026-08-25 ~05:42, retired the same day for the AGENT model). See [[frank-dragoon]]. +--- diff --git a/serverAppConfigs-docs/LLM/ling-3.0-flash-trial.md b/serverAppConfigs-docs/LLM/ling-3.0-flash-trial.md index b9db9dc..331ae85 100644 --- a/serverAppConfigs-docs/LLM/ling-3.0-flash-trial.md +++ b/serverAppConfigs-docs/LLM/ling-3.0-flash-trial.md @@ -1,9 +1,9 @@ --- type: reference -description: "`bailingmoe3-support` (commit `db4480bc8`, build b10411) at `/home/dank/llama.cpp-ling`." +description: "Trial of a fork build for a new mixture-of-experts architecture: what loaded, what the harness refused, and the verdict." tags: - llm -timestamp: 2026-09-03 +timestamp: 2026-10-19 --- # Ling-3.0-flash Q6_K Trial on Dankclaw @@ -109,6 +109,13 @@ re-enable `frank-dragoon-ling` from the repo unit - The heap is effectively full (98.7 of 111.2 GiB). Do not raise the context much without checking `--list-devices` free MiB. +## The other path: vLLM-ROCm (never shipped) + +A deployment plan existed for `jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native` +(INT4 W4A16, ~62 GB) served by vLLM-ROCm on Dankclaw. It never shipped; the +llama.cpp Q6_K fork trial above is what actually ran. The decision table the +plan carried is in git history. + ## See Also - [[frank-dragoon]] diff --git a/serverAppConfigs-docs/LLM/ling-3.0-flash-vllm-plan.md b/serverAppConfigs-docs/LLM/ling-3.0-flash-vllm-plan.md deleted file mode 100644 index 53d8e0d..0000000 --- a/serverAppConfigs-docs/LLM/ling-3.0-flash-vllm-plan.md +++ /dev/null @@ -1,22 +0,0 @@ ---- -type: guide -description: "This page was the plan for a vLLM-ROCm deployment of `jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native` (INT4 W4A16, ~62 GB) on" -tags: - - llm - - vllm - - rocm -timestamp: 2026-09-03 ---- - -# Ling-3.0-flash on Dankclaw — vLLM-ROCm Deployment Plan (never shipped) - -This page was the plan for a vLLM-ROCm deployment of -`jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native` (INT4 W4A16, ~62 GB) on -Dankclaw. The deployment never shipped. The actual Ling work used the -llama.cpp Q6_K fork trial instead: [[ling-3.0-flash-trial]]. - -The trial service (`frank-dragoon-ling`) was retired and disabled under the boot -policy on 2026-08-17, in the same redo that renamed `frank-dragoon` to the -Qwen3.8-27B PP head ([[frank-redo-2026-08-17]]). Two fleet generations later the -name belongs to the halogen flash-server endpoint ([[frank-dragoon]]). See git -history for the full decision table. diff --git a/serverAppConfigs-docs/LLM/qwen3.8-spec-matrix.md b/serverAppConfigs-docs/LLM/qwen3.8-spec-matrix.md index 68f70b0..359cb91 100644 --- a/serverAppConfigs-docs/LLM/qwen3.8-spec-matrix.md +++ b/serverAppConfigs-docs/LLM/qwen3.8-spec-matrix.md @@ -1,6 +1,6 @@ --- type: reference -description: "This page records the speculative-decoding benchmark history and the" +description: "The speculative-decoding benchmark history for this endpoint, arm by arm, and what each arm ruled out." tags: - llm - speculative diff --git a/serverAppConfigs-docs/LLM/qwen3.8-strix-optimizations.md b/serverAppConfigs-docs/LLM/qwen3.8-strix-optimizations.md index e058b37..fc67dbc 100644 --- a/serverAppConfigs-docs/LLM/qwen3.8-strix-optimizations.md +++ b/serverAppConfigs-docs/LLM/qwen3.8-strix-optimizations.md @@ -1,6 +1,6 @@ --- type: reference -description: "This page answers one question: **how do we make Qwen3.8-27B serve faster on" +description: "How a served checkpoint got faster on the Strix Halo iGPU: the lever ladder and what each lever actually bought." tags: - llm timestamp: 2026-09-03 diff --git a/serverAppConfigs-docs/LLM/swift15-campaign-worklog.md b/serverAppConfigs-docs/LLM/swift15-campaign-worklog.md index 1cb81cd..1042c26 100644 --- a/serverAppConfigs-docs/LLM/swift15-campaign-worklog.md +++ b/serverAppConfigs-docs/LLM/swift15-campaign-worklog.md @@ -1,3 +1,13 @@ +--- +type: "reference" +title: "Swift1.5 Campaign Worklog" +description: "Dated worklog of the campaign that converted and served a Swift checkpoint on the radiance endpoint, arm by arm." +tags: + - llm + - campaign +timestamp: 2026-10-19 +--- + # Swift1.5 w4dn8 Campaign — Work Log Campaign branch `swift15-radiance-w4dn8`. Goal: convert diff --git a/serverAppConfigs-docs/LLM/ubatch-benchmark-large.md b/serverAppConfigs-docs/LLM/ubatch-benchmark-large.md index b228e18..a1a0ed8 100644 --- a/serverAppConfigs-docs/LLM/ubatch-benchmark-large.md +++ b/serverAppConfigs-docs/LLM/ubatch-benchmark-large.md @@ -1,7 +1,7 @@ --- -type: benchmark plan +type: reference title: frank-dragoon ubatch sweep -description: Standardized ubatch sweep for the active kingjones Qwen3.8 ROCmFPX deployment. +description: "The wide ubatch sweep for a deployed ROCmFPX shape, run through the standardized harness." tags: - llama.cpp - rocm diff --git a/serverAppConfigs-docs/LLM/vLLM Boot Forensics and Config Interrogation.md b/serverAppConfigs-docs/LLM/vLLM Boot Forensics and Config Interrogation.md index 741ef0a..cb97a9c 100644 --- a/serverAppConfigs-docs/LLM/vLLM Boot Forensics and Config Interrogation.md +++ b/serverAppConfigs-docs/LLM/vLLM Boot Forensics and Config Interrogation.md @@ -1,7 +1,7 @@ --- type: reference title: vLLM Boot Forensics and Config Interrogation -description: Capturing a vLLM container boot failure's real root cause from journald (the container log is already gone), classifying the failure, and interrogating a live endpoint's true config and client contracts. +description: "Recovering a container boot failure's real cause from journald, and interrogating a live endpoint's true config." tags: [vllm, forensics, journalctl, podman, config, contracts] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/LLM/vLLM Porting and Compile Triage.md b/serverAppConfigs-docs/LLM/vLLM Porting and Compile Triage.md index b0eeddf..fab8306 100644 --- a/serverAppConfigs-docs/LLM/vLLM Porting and Compile Triage.md +++ b/serverAppConfigs-docs/LLM/vLLM Porting and Compile Triage.md @@ -1,7 +1,7 @@ --- type: reference title: vLLM Porting and Compile Triage -description: Forward-porting a model architecture to a new vLLM major (additive-only discipline, gap enumeration, shims that survive worker spawn) and the torch.compile/dynamo break triage, including the EAGLE3/DFlash2 aux-hidden-state protocol port rules. +description: "Forward-porting a model architecture: additive-only discipline, shims that survive worker spawn, compile triage." tags: [vllm, porting, torch-compile, dynamo, eagle3, dflash, glm] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/LLM/vllm_research_findings.md b/serverAppConfigs-docs/LLM/vllm_research_findings.md index cf1963a..dc918d6 100644 --- a/serverAppConfigs-docs/LLM/vllm_research_findings.md +++ b/serverAppConfigs-docs/LLM/vllm_research_findings.md @@ -1,6 +1,6 @@ --- type: reference -description: "vLLM v0.26.0 (latest stable, released July 25, 2026) supports DFlash speculative decoding with the `--spec-method dflash` and `--spec-model` flags." +description: "Upstream support survey for speculative decoding: which flags, releases and pull requests carry each method." tags: - llm - vllm @@ -10,7 +10,17 @@ timestamp: 2026-08-28 # Research: vLLM Parameters for Specified Configuration ## Summary -vLLM v0.26.0 (latest stable, released July 25, 2026) supports DFlash speculative decoding with the `--spec-method dflash` and `--spec-model` flags. KV cache dtype `q8_0` is **not supported** — only fp8 variants and turboquant options are available. vLLM does not have a dedicated MoE expert CPU offload flag; general CPU offloading via `--cpu-offload-gb` is the closest option. The `--max-num-seqs` flag is confirmed. Podman GPU access requires `--gpus all` with the NVIDIA container toolkit. +vLLM v0.26.0 (released 2026-07-25, the current stable when this survey was +gathered on 2026-08-28; PyPI shows v0.32.0 as the latest on 2026-10-19, so +re-check this page before acting on it) supports DFlash speculative decoding. +The preferred CLI is the `--speculative-config` JSON object; `--spec-method` +and `--spec-model` are the per-field aliases defined in +`vllm/engine/arg_utils.py`. KV cache dtype `q8_0` is **not supported** — only +fp8 variants and turboquant options are available. vLLM does not have a +dedicated MoE expert CPU offload flag; general CPU offloading via +`--cpu-offload-gb` is the closest option. The `--max-num-seqs` flag is +confirmed. Podman GPU access requires `--gpus all` with the NVIDIA container +toolkit. ## Findings diff --git a/serverAppConfigs-docs/LLM/x4-pair-experiment.md b/serverAppConfigs-docs/LLM/x4-pair-experiment.md index 72fc264..d54348d 100644 --- a/serverAppConfigs-docs/LLM/x4-pair-experiment.md +++ b/serverAppConfigs-docs/LLM/x4-pair-experiment.md @@ -1,7 +1,7 @@ --- type: reference title: The x4 pair — Flash-Next across the 3090 and the Strix Halo iGPU -description: What mainline llama.cpp can and cannot do with a 3090 + Strix Halo combination, measured 2026-09-16 on Dankclaw — the tensor inventory that decides any split, the single flag that kills per-tensor placement across devices (-ot), the two-device layer splits that DO work (pure-Vulkan and cross-vendor CUDA+Vulkan, +7 % decode at 64k depth), and the harness-parsing trap that made the first round of measurements meaningless. +description: "Measured limits of a consumer card plus APU pair: the tensor inventory, the flag that kills cross-device placement." tags: - llm - experiment diff --git a/serverAppConfigs-docs/Operations/Container Host Reachability.md b/serverAppConfigs-docs/Operations/Container Host Reachability.md index edda0aa..497c495 100644 --- a/serverAppConfigs-docs/Operations/Container Host Reachability.md +++ b/serverAppConfigs-docs/Operations/Container Host Reachability.md @@ -1,7 +1,7 @@ --- type: reference title: Container Host Reachability -description: The two podman/netavark reachability quirks on the fleet — containers cannot dial unpublished host ports (LAN IP or loopback), and netns membership decides what "reachable" means. Probes that prove it, fixes, and the pasta port gate. +description: "Two reachability quirks: containers cannot dial unpublished host ports, and netns membership decides reachability." tags: [podman, netavark, networking, rootless, reachability, caddy] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/Operations/Fleet Audit and Posture.md b/serverAppConfigs-docs/Operations/Fleet Audit and Posture.md index 170cb3e..3f8339a 100644 --- a/serverAppConfigs-docs/Operations/Fleet Audit and Posture.md +++ b/serverAppConfigs-docs/Operations/Fleet Audit and Posture.md @@ -1,7 +1,7 @@ --- type: reference title: Fleet Audit and Posture -description: Three standing audits — monitoring completeness (what should exist vs what is merely down), vhost proxy health decoding, and the behavioral security posture audit; plus the stash supersession check before dropping old work. +description: "Three standing audits: monitoring completeness, vhost proxy health, and behavioural security posture." tags: [audit, security, prometheus, caddy, monitoring, git] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/Operations/Fleet Ops Playbook.md b/serverAppConfigs-docs/Operations/Fleet Ops Playbook.md index ce7044a..94448b8 100644 --- a/serverAppConfigs-docs/Operations/Fleet Ops Playbook.md +++ b/serverAppConfigs-docs/Operations/Fleet Ops Playbook.md @@ -1,7 +1,7 @@ --- type: reference title: Fleet Ops Playbook -description: Incident-shaped fleet procedures — LLM endpoint down with an abandoned GPU holder, scratch vLLM boot windows (stop-first law), iterating on a fleet service before commit, incident-window reconstruction, and waking a sleeping host. +description: "Incident-shaped procedures: endpoint down with a GPU holder, stop-first bench windows, service iteration, waking a host." tags: [operations, incidents, gpu, systemd, wol, forensics] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/Operations/Payload and Disk Reclaim.md b/serverAppConfigs-docs/Operations/Payload and Disk Reclaim.md index a19a0e6..c81c57b 100644 --- a/serverAppConfigs-docs/Operations/Payload and Disk Reclaim.md +++ b/serverAppConfigs-docs/Operations/Payload and Disk Reclaim.md @@ -1,7 +1,7 @@ --- type: reference title: Payload and Disk Reclaim -description: Standing procedure for disk pressure on a fleet host — measure all buckets first, classify against the runbook index, prove nothing live references the payload, stage to trash with fingerprints, purge only on approval; plus the purge-and-refresh protocol with fetch verification. +description: "Disk pressure procedure: measure every bucket, classify, prove nothing references the payload, stage, purge on approval." tags: [operations, disk, podman, payloads, huggingface, trash] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/Operations/Remote Job Runner.md b/serverAppConfigs-docs/Operations/Remote Job Runner.md index fe428df..1a21977 100644 --- a/serverAppConfigs-docs/Operations/Remote Job Runner.md +++ b/serverAppConfigs-docs/Operations/Remote Job Runner.md @@ -1,7 +1,7 @@ --- type: reference title: Remote Job Runner -description: Detached long-job execution on fleet hosts with sentinel files (job.sh) — why ssh tail-watchers fail structurally, the script, its invariants, verification, and reporting discipline. +description: "Detached long jobs with sentinel files: why tail-watchers fail structurally, the script, its invariants, verification." tags: [operations, ssh, jobs, sentinel, detached] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/Operations/SSH and Remote Access.md b/serverAppConfigs-docs/Operations/SSH and Remote Access.md index 1ec5583..ab7f76f 100644 --- a/serverAppConfigs-docs/Operations/SSH and Remote Access.md +++ b/serverAppConfigs-docs/Operations/SSH and Remote Access.md @@ -1,7 +1,7 @@ --- type: reference title: SSH and Remote Access -description: Passwordless SSH bootstrap without sshpass/expect, remote execution against fish login shells, and diagnosing a frozen GUI from an SSH session. +description: "Passwordless SSH bootstrap without sshpass, remote execution against fish login shells, and frozen-GUI triage." tags: [ssh, remote, fish, gui, diagnosis] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/Operations/index.md b/serverAppConfigs-docs/Operations/index.md index ec58fa1..a77c91d 100644 --- a/serverAppConfigs-docs/Operations/index.md +++ b/serverAppConfigs-docs/Operations/index.md @@ -1,28 +1,15 @@ ---- -type: index -description: Operational procedures for the fleet — job runners, disk and payload reclaim, container reachability, audits, SSH craft, and incident playbooks. -tags: [operations, index] -timestamp: 2026-10-05T00:00:00-07:00 ---- - # Operations Procedures for operating the fleet. The LLM-specific pages live under [LLM](../LLM/index.md); infrastructure and architecture under the `../Infrastructure/` and `../Architecture/` folders. -- [[Remote Job Runner]] — detached long-job execution with sentinel files; - why ssh tail-watchers fail -- [[Payload and Disk Reclaim]] — measure-classify-prove-stage-ask disk - procedure, plus the purge-and-refresh protocol -- [[Container Host Reachability]] — netavark/pasta reachability quirks and - the pasta port gate -- [[Fleet Ops Playbook]] — endpoint-down GPU-holder reclaim, scratch-boot - stop-first, pre-commit iteration, incident-window reconstruction, WoL -- [[Fleet Audit and Posture]] — monitoring gaps, vhost probes, security - posture, whole-fleet audit, bulk redaction, stash audits -- [[SSH and Remote Access]] — passwordless bootstrap without sshpass, fish - login shells, GUI-hang diagnosis over ssh +- [[Remote Job Runner]] — Detached long jobs with sentinel files: why tail-watchers fail structurally, the script, its invariants, verification. +- [[Payload and Disk Reclaim]] — Disk pressure procedure: measure every bucket, classify, prove nothing references the payload, stage, purge on approval. +- [[Container Host Reachability]] — Two reachability quirks: containers cannot dial unpublished host ports, and netns membership decides reachability. +- [[Fleet Ops Playbook]] — Incident-shaped procedures: endpoint down with a GPU holder, stop-first bench windows, service iteration, waking a host. +- [[Fleet Audit and Posture]] — Three standing audits: monitoring completeness, vhost proxy health, and behavioural security posture. +- [[SSH and Remote Access]] — Passwordless SSH bootstrap without sshpass, remote execution against fish login shells, and frozen-GUI triage. ## Related - [Observability Provisioning](../Infrastructure/Observability%20Provisioning.md) diff --git a/serverAppConfigs-docs/Research/AI Job Search and Resume Tools.md b/serverAppConfigs-docs/Research/AI Job Search and Resume Tools.md index cfdedc9..c3572e9 100644 --- a/serverAppConfigs-docs/Research/AI Job Search and Resume Tools.md +++ b/serverAppConfigs-docs/Research/AI Job Search and Resume Tools.md @@ -1,6 +1,6 @@ --- -type: research -description: "Self-hosted AI job-search and resume-formatting applications: the 2026 landscape, local-LLM fit, and a recommended homelab stack." +type: reference +description: "Self-hosted job-search and resume tools: the landscape, local-LLM fit, and the recommended homelab stack." tags: - research - job-search diff --git a/serverAppConfigs-docs/Research/ComfyUI ROCm and Vulkan.md b/serverAppConfigs-docs/Research/ComfyUI ROCm and Vulkan.md index 3abcb17..c33c0f8 100644 --- a/serverAppConfigs-docs/Research/ComfyUI ROCm and Vulkan.md +++ b/serverAppConfigs-docs/Research/ComfyUI ROCm and Vulkan.md @@ -1,6 +1,6 @@ --- type: reference -description: "Research into whether ComfyUI can use GPU acceleration — not just CPU — on the AMD Strix Halo APU (gfx1151) in Dankclaw." +description: "Whether ComfyUI can use the Strix Halo APU at all, and which backend path actually reaches the GPU." tags: - research - rocm diff --git a/serverAppConfigs-docs/Research/Llama Cpp Code Diff.md b/serverAppConfigs-docs/Research/Llama Cpp Code Diff.md index cfb62ca..3c0e73d 100644 --- a/serverAppConfigs-docs/Research/Llama Cpp Code Diff.md +++ b/serverAppConfigs-docs/Research/Llama Cpp Code Diff.md @@ -1,7 +1,7 @@ --- -type: research +type: reference title: Llama.cpp Code Diff -description: Code-level diff of mainline llama.cpp, charlie12345/ROCmFPX, and ik_llama.cpp with a focus on quantization code, ggml type numbers, and the differences that matter for Dankserver (RTX 3090) and Dankclaw (Strix Halo). +description: "Code-level diff of the three llama.cpp trees here: quantization code, type numbers, and what differs for our cards." tags: - llama.cpp - quantization diff --git a/serverAppConfigs-docs/Research/Llama Cpp Runtimes.md b/serverAppConfigs-docs/Research/Llama Cpp Runtimes.md index 054aad9..f3af259 100644 --- a/serverAppConfigs-docs/Research/Llama Cpp Runtimes.md +++ b/serverAppConfigs-docs/Research/Llama Cpp Runtimes.md @@ -1,7 +1,7 @@ --- -type: research +type: reference title: Llama.cpp Runtimes -description: Compared mainline llama.cpp, charlie12345/ROCmFPX, and ik_llama.cpp — positioning, quantization formats, code-level differences, and fit for Dankserver (RTX 3090) and Dankclaw (Strix Halo). +description: "The llama.cpp trees on this host compared: positioning, quantization formats, and fit for each machine." tags: - llama.cpp - quantization diff --git a/serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md b/serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md index bcb111a..5637fbe 100644 --- a/serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md +++ b/serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md @@ -1,7 +1,7 @@ --- -type: research +type: reference title: Local TTS and Voice Cloning -description: How to create a new local TTS voice from sample files in oh-my-pi, the 2026 open-weight voice-cloning landscape, the measured CPU engine A/B behind frank-tts and the arm A decision, the full local AI audio stack, and a recommended homelab deployment. +description: "Creating a local voice from sample files: the open-weight cloning landscape, the measured engine A/B, the audio stack." tags: - research - tts diff --git a/serverAppConfigs-docs/Research/Most Popular Self-Hosted AI Apps.md b/serverAppConfigs-docs/Research/Most Popular Self-Hosted AI Apps.md index 1610366..195bc73 100644 --- a/serverAppConfigs-docs/Research/Most Popular Self-Hosted AI Apps.md +++ b/serverAppConfigs-docs/Research/Most Popular Self-Hosted AI Apps.md @@ -1,6 +1,6 @@ --- -type: research -description: "The 15 most popular self-hosted AI applications of June–September 2026, ranked by GitHub stars, with homelab fit and adoption best practices." +type: reference +description: "The most-starred self-hosted AI applications of the year so far, with homelab fit and adoption notes." tags: - research - self-hosted diff --git a/serverAppConfigs-docs/Research/PICO-8 Asset Standards.md b/serverAppConfigs-docs/Research/PICO-8 Asset Standards.md index 161745b..ae67d02 100644 --- a/serverAppConfigs-docs/Research/PICO-8 Asset Standards.md +++ b/serverAppConfigs-docs/Research/PICO-8 Asset Standards.md @@ -1,7 +1,7 @@ --- type: reference title: PICO-8 Asset Standards -description: Production-grade standards for PICO-8 asset sheets — palette discipline, single perspective and light direction, detail flow and hierarchy, sheet organization, and the cohesion checklist. +description: "Asset-sheet standards for PICO-8: palette discipline, one light direction, detail hierarchy, cohesion checklist." tags: [pico8, game-dev, pixel-art, standards] timestamp: 2026-10-05T00:00:00-07:00 --- diff --git a/serverAppConfigs-docs/Research/SGLang.md b/serverAppConfigs-docs/Research/SGLang.md index 82adf06..102ecb3 100644 --- a/serverAppConfigs-docs/Research/SGLang.md +++ b/serverAppConfigs-docs/Research/SGLang.md @@ -1,7 +1,7 @@ --- -type: research +type: reference title: SGLang -description: What SGLang is, how it works (RadixAttention, paged KV, overlap scheduler, structured output), how it compares to llama.cpp and vLLM, and whether it fits Dankserver (RTX 3090) and Dankclaw (Strix Halo). +description: "What SGLang is, how it schedules and caches, how it compares to the runtimes here, and whether it fits this fleet." tags: - sglang - serving diff --git a/serverAppConfigs-docs/Research/Service Research Brief.md b/serverAppConfigs-docs/Research/Service Research Brief.md index b6075a4..d62aea6 100644 --- a/serverAppConfigs-docs/Research/Service Research Brief.md +++ b/serverAppConfigs-docs/Research/Service Research Brief.md @@ -1,6 +1,6 @@ --- type: reference -description: "Consolidated research from the `frank-zealot` research subagents (2026-08-14) plus the pi agent knowledge base." +description: "Consolidated upstream research on the fleet's LLM runtimes, dated to when it was gathered." tags: - research timestamp: 2026-09-03 diff --git a/serverAppConfigs-docs/Research/index.md b/serverAppConfigs-docs/Research/index.md index 9994974..560d980 100644 --- a/serverAppConfigs-docs/Research/index.md +++ b/serverAppConfigs-docs/Research/index.md @@ -5,32 +5,35 @@ findings doc; the project root index ([serverAppConfigs-docs/index.md](../index. ## LLM Inference -- [[Llama Cpp Runtimes]] — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: positioning, quantization, server fit -- [[Llama Cpp Code Diff]] — code-level quantization diff + comparison table + server recommendations -- [[SGLang]] — the structured-generation serving engine (llama.cpp/vLLM comparison, our- servers fit) +- [[Llama Cpp Runtimes]] — The llama.cpp trees on this host compared: positioning, quantization formats, and fit for each machine. +- [[Llama Cpp Code Diff]] — Code-level diff of the three llama.cpp trees here: quantization code, type numbers, and what differs for our cards. +- [[SGLang]] — What SGLang is, how it schedules and caches, how it compares to the runtimes here, and whether it fits this fleet. ## Image / Media -- [[ComfyUI ROCm and Vulkan]] — live-verified GPU path for ComfyUI on Strix Halo -- [[Lemonade and sd.cpp]] — the ggml-family image-generation alternative -- [[Jellyfin]] — media server research brief +- [[ComfyUI ROCm and Vulkan]] — Whether ComfyUI can use the Strix Halo APU at all, and which backend path actually reaches the GPU. +- [[Lemonade and sd.cpp]] — Why Lemonade Server can do image generation on Strix Halo when ComfyUI cannot. ## Audio / TTS -- [[Local TTS and Voice Cloning]] — creating a local voice from samples, the 2026 cloning landscape, the full audio stack, and a recommended homelab deployment +- [[Local TTS and Voice Cloning]] — Creating a local voice from sample files: the open-weight cloning landscape, the measured engine A/B, the audio stack. ## Career / Job Search -- [[AI Job Search and Resume Tools]] — self-hosted AI job-search + resume-formatting apps, local-LLM fit, recommended stack +- [[AI Job Search and Resume Tools]] — Self-hosted job-search and resume tools: the landscape, local-LLM fit, and the recommended homelab stack. ## Self-Hosted Landscape -- [[Most Popular Self-Hosted AI Apps]] — the 15 most popular self-hosted AI apps of Jun–Sep 2026, homelab fit, adoption best practices (HTML report: `popular-self-hosted-ai-apps.html`) +- [[Most Popular Self-Hosted AI Apps]] — The most-starred self-hosted AI applications of the year so far, with homelab fit and adoption notes. + +## Games / Creative + +- [[PICO-8 Asset Standards]] — Asset-sheet standards for PICO-8: palette discipline, one light direction, detail hierarchy, cohesion checklist. ## Backups -- [[Backblaze Integration]] — off-site backup plan (future) +- [[Backblaze Integration]] — Off-site backup plan for the bind-mounted data on both hosts. ## Legacy -- [[Service Research Brief]] — consolidated subagent research behind these pages +- [[Service Research Brief]] — Consolidated upstream research on the fleet's LLM runtimes, dated to when it was gathered. diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 9c77d48..439a818 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -11,125 +11,120 @@ is a locator: one row, one clause, no numbers. | Page | Owns | |---|---| -| [[Frank TTS]] | frank-tts is the fleet's cloned-voice speech endpoint on Dankclaw :7772 — the one voice surface consumers address, in… | -| [[frank-archon]] | The one frank-archon page: the live radiance shape and harness contract (read off scripts/frank-archon.env), the Eras… | -| [[frank-dragoon]] | The one runbook for frank-dragoon, the Strix Halo iGPU endpoint (192.168.0.143:7770): current shape from the env,… | -| [[frank-zealot]] | The one frank-zealot reference: the live shape on Dankclaw's RTX 3090 (:7771) exactly as scripts/frank-zealot.env… | +| [[frank-archon]] | The radiance endpoint: live shape and harness contract, every era with its rollback path, and the laws it taught. | +| [[frank-dragoon]] | The iGPU endpoint: live shape, every era with its rollback path, the laws it paid for, and the files that own it. | +| [[frank-zealot]] | The RTX 3090 endpoint: live shape exactly as the env ships it, every era with its rollback path, and its card laws. | ## Retired endpoints -These services no longer exist, so no live page owns them; rollback provenance -is inside each page. Every payload swap of a LIVE endpoint is an Eras row on -that endpoint's page, not a page. - | Page | Owns | |---|---| -| [[frank-medium-batch-ubatch]] | llama-bench sweep of --batch-size against --ubatch-size on frank-medium (2x Radeon AI PRO R9700, layer split) at the… | -| [[frank-medium-llamacpp-dflash2]] | The former ACTIVE frank-medium deployment since 2026-09-16 (model swapped 2026-09-19) — mainline llama.cpp on… | -| [[frank-medium-model-swap]] | RETIRED-era swap procedure for the frank-medium endpoint (`:7770` is now [[frank-archon]]) — the… | -| [[frank-medium-radiance-mxfp4]] | RETIRED frank-medium era runbook — vllm-radiance serving hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 (ParoQuant MXFP6 W4A8,… | -| [[frank-medium-rocmfpx]] | charlie12345/ROCmFPX llama.cpp serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF as frank-medium on Dankserver… | -| [[frank-npu]] | The retired FastFlowLM endpoint on Dankclaw's XDNA2 NPU — what it served, why it died on 2026-09-11 with the… | -| [[frank-redo-2026-08-17]] | This page records the full redo of the inference fleet on 2026-08-17. | -| [[ikllama-ornith-dankserver]] | ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-zealot on the RTX 3090. Q8_0 model, KV K default f16 / V… | -| [[kingjones-q8agent-rocmfpx]] | **Status (historical, 2026-09-13):** was to deploy on Dankclaw. | -| [[qwen3.8-spec-matrix]] | This page records the speculative-decoding benchmark history and the | +| [[dealign-ornith-crack]] | Retired era: a mixture-of-experts checkpoint served with CPU expert offload and n-gram drafting on a single card. | +| [[frank-medium]] | Locator for the retired Dankserver LLM endpoint: which era page owns which payload, and where the dates disagree. | +| [[frank-medium-llamacpp-dflash2]] | Retired era: the layer-split llama.cpp shape with a drafter, its ladder measurements, and the balance it settled on. | +| [[frank-medium-model-swap]] | Retired-era swap procedure: the fetch toolchain, the entrypoint contract, the capability battery, the rollback path. | +| [[frank-medium-radiance-mxfp4]] | Retired era: the tensor-parallel radiance shape on two cards, its quant-class swap, and the on-disk rollback payload. | +| [[frank-medium-rocmfpx]] | Retired era: the ROCmFPX fork serving an agent-tuned checkpoint on the two-card box, and the loader pin it required. | +| [[frank-npu]] | The retired NPU endpoint: what it served, why it died with the IOMMU decision, and where every dependent route went. | +| [[frank-redo-2026-08-17]] | The day the whole inference fleet was rebuilt: what came down, what replaced it, and why it was done that way. | +| [[ikllama-ornith-dankserver]] | Retired era: the ik_llama fork with a quantized value cache and CPU expert offload, and the build flags it required. | +| [[kingjones-q8agent-rocmfpx]] | Historical: a checkpoint and loader pairing staged for this host and never deployed on it. | -## Dankclaw services (192.168.0.143) +## Dankclaw services | Page | Owns | |---|---| | [[ComfyUI]] | ComfyUI is a node-based AI image generation engine. | -| [[ComfyUI Qwen-Image Workflows]] | Qwen-Image 2.1 workflows on Dankclaw: two proven txt2img workflows (art + legible text), the API recipe, and which… | +| [[ComfyUI Qwen-Image Workflows]] | Proven Qwen-Image text-to-image workflows on Dankclaw: the recipes, the API call shape, and which models to add next. | | [[Dankclaw Stack]] | Service inventory and routing overview for the Dankclaw AI, development, and observability host. | -| [[Frank TTS Engine]] | frank-tts-engine is the arm A TTS engine on Dankclaw: qwentts.cpp's resident tts-server with the 1.7 B base talker,… | +| [[Frank TTS]] | The fleet's cloned-voice speech endpoint and the one voice surface consumers address; wired as the harness speech role. | +| [[Frank TTS Engine]] | The engine behind the fleet's speech endpoint: its runtime, private bridge, and the surface it publishes. | | [[Gitea]] | Gitea deployment on Dankclaw with its web UI reverse-proxied by the canonical Caddy service. | | [[Gluetun]] | Gluetun is a lightweight VPN client container. | | [[Hermes]] | Hermes is the self-improving AI agent from Nous Research. | | [[Hermes-Dashboard]] | The Hermes web dashboard is a browser UI for managing the agent. | -| [[Hermes-WebUI]] | The hermes-webui container on Dankclaw — the API server that the Hermex iPhone app talks to, reachable over an SSH… | -| [[Marreta]] | The content-cleaning proxy (manualdousuario/marreta, Laravel/serversideup fpm-nginx) deployed as a rootless Quadlet… | -| [[Open Notebook]] | open-notebook is the fleet's podcast generator on Dankclaw: notes and sources in, a multi-speaker episode out —… | +| [[Hermes-WebUI]] | The iPhone companion app for Hermes on Dankclaw and the ways it is reached from the LAN. | +| [[Marreta]] | The content-cleaning proxy SearXNG borrows egress from: a quadlet pair in the VPN netns behind the guard chain. | +| [[Open Notebook]] | The fleet's podcast generator: notes in, multi-speaker episode out, script and voices both served in-house. | | [[OpenWebUI]] | OpenWebUI is a self-hosted chat interface for LLMs. | | [[Postgres]] | PostgreSQL is the database backend for Gitea. | | [[SearXNG]] | SearXNG is a privacy-respecting metasearch engine. | | [[SearXNG Providers]] | How Google and other upstream search engines get blocked behind the Gluetun | -| [[Seerr]] | Seerr is the unified request manager for the *arr stack (the maintained `seerr-team/seerr` successor to the archived… | -| [[Swap Configuration]] | The host swapfile on Dankclaw — 64 GiB at /swap.img on ext4, fstab-driven via the generated swap.img.swap unit.… | +| [[Seerr]] | The request manager for the *arr stack — the maintained successor to the archived Overseerr line. | +| [[Swap Configuration]] | The host swapfile: why it is sized as it is, the resize ordering that avoids an OOM, and the swappiness caveat. | | [[Valkey]] | Valkey is the cache and limiter store for SearXNG. | -| [[Voice Packs]] | How a frank-tts voice pack is built on Dankclaw: the three files every pack must carry, the 10-second source-material… | +| [[Voice Packs]] | How a voice pack is built: the files every pack must carry, the source-material gate, adding a voice, engine-bump drift. | -## Dankserver services (192.168.0.235) +## Dankserver services | Page | Owns | |---|---| | [[Calibre]] | Calibre is the e-book library manager that backs Calibre-Web. | | [[Calibre-Web]] | Calibre-Web is a web application that browses, reads, and downloads eBooks from an existing Calibre library database. | | [[Copyparty]] | Copyparty is a lightweight file server and media streamer that runs as a single Python binary. | -| [[Dankserver Stack]] | Dankserver (192.168.0.235) runs the media and library stack. | +| [[Dankserver Stack]] | Service inventory and routing overview for the media and library host. | | [[Entropy Volume]] | The Dankserver entropy volume is a RAID-backed LUKS filesystem mounted at | -| [[Jellyfin]] | Jellyfin media server on Dankserver — quadlet unit, entropy-volume data layout, ports, client access, and… | -| [[mariadb]] | MariaDB 11 database backend for RomM on Dankserver — quadlet unit, network alias, env-file secret handling, and… | +| [[Jellyfin]] | Jellyfin on Dankserver: quadlet unit, entropy-volume data layout, ports, client access, hardware-transcode status. | +| [[mariadb]] | The MariaDB backend for RomM: quadlet unit, network alias, env-file secret handling, and the backup path. | | [[Plex]] | Plex Media Server is a **containerized media server on Dankserver**: a podman | -| [[redis]] | Redis 7-alpine cache and task-queue backend for RomM on Dankserver — quadlet unit, network alias topology, and backup… | +| [[redis]] | The Redis cache and task queue for RomM: quadlet unit, network alias topology, and the backup path. | | [[RomM]] | RomM (ROM Manager) is a self-hosted ROM library manager and player. | -| [[RomM-Scraping]] | This guide covers metadata providers, cover art, ROM naming conventions, emulator integration, and the scraping… | +| [[RomM-Scraping]] | Metadata providers, cover art, ROM naming and the scraping workflow behind the RomM library. | | [[Tautulli]] | Tautulli is the statistics, monitoring, and notification companion for | -## FrankBook-Pro (192.168.0.112) +## FrankBook-Pro | Page | Owns | |---|---| -| [[ARR-Stack]] | The *arr automation stack (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) — FrankBook-Pro only, every… | +| [[ARR-Stack]] | The *arr automation stack on FrankBook-Pro, every container on the VPN netns: topology, wiring, and gotchas. | | [[lidarr]] | Lidarr music manager in the *arr stack on FrankBook-Pro — live quadlet deployment, ports, volumes, and wiring facts. | -| [[prowlarr]] | Prowlarr indexer-management layer for the *arr stack on FrankBook-Pro — feed routes, sync model, API surface, and… | -| [[qbittorrent]] | qBittorrent download client for the *arr stack — pinned 4.6.7 on FrankBook-Pro, shares the Gluetun WireGuard netns,… | +| [[prowlarr]] | Indexer management for the *arr stack: feed routes, the sync model, the API surface, and verified live gotchas. | +| [[qbittorrent]] | The download client for the *arr stack: pinned image, shared WireGuard netns, WebUI credential recipe. | | [[radarr]] | Radarr movie manager in the *arr stack on FrankBook-Pro — live quadlet deployment, ports, volumes, and wiring facts. | -| [[readarr]] | Readarr ebook/audiobook manager in the *arr stack on FrankBook-Pro — retired upstream, pinned image, parked qB… | +| [[readarr]] | The ebook and audiobook manager in the *arr stack: retired upstream, pinned image, parked download client. | | [[sonarr]] | Sonarr TV manager in the *arr stack on FrankBook-Pro — live quadlet deployment, ports, volumes, and wiring facts. | ## Workstations | Page | Owns | |---|---| -| [[Noctalia and Niri]] | The noctalia v5 desktop shell on niri (workstation + blankpad) — config layers, remote IPC over ssh, custom-palette… | +| [[Noctalia and Niri]] | The noctalia shell on niri across workstations: config layers, remote IPC, custom palettes, output rotation and pinning. | ## Cross-host infrastructure | Page | Owns | |---|---| -| [[Alertmanager]] | Alertmanager receives firing rules from Prometheus, groups and dedupes them, and forwards each group as a webhook to… | +| [[Alertmanager]] | Receives firing rules from Prometheus, groups and dedupes them, and forwards each group to the host-side Signal relay. | | [[Alloy]] | Alloy is Grafana's log and telemetry shipper. | | [[Anubis]] | Anubis is the bot-challenge proxy that fronts the guarded public vhosts on | | [[Authentik]] | Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the | -| [[authentik-postgres]] | The PostgreSQL 16 database for Authentik — the homelab's SSO / | -| [[authentik-server]] | The Authentik server — the core, the admin API, and the embedded Caddy outpost. | -| [[authentik-worker]] | The Authentik worker — background tasks: event processing, provider | +| [[Authentik Outpost Reference]] | Outpost, proxy-provider, OAuth2, LDAP and flow configuration for the homelab's Authentik SSO. | +| [[Authentik Processes]] | The three core Authentik containers — database, server and worker — with their units, images, roles and probes. | | [[Caddy]] | Caddy is the single reverse proxy for both hosts. | | [[FrankBook-Pro]] | FrankBook-Pro — the Mac mini running the *arr stack + qBittorrent inside a podman machine VM, gluetun-gated | | [[Grafana]] | Grafana visualizes the metrics that Prometheus collects from every service. | -| [[Homepage]] | A password-protected web site on Dankclaw lists the services that are up, with a short description of each and a link… | +| [[Homepage]] | The password-protected service list on Dankclaw: what each card shows and where its data comes from. | | [[Homepage Design Grilling]] | The decision log for the homepage design. | -| [[Homepage Status Probes]] | The homepage TUI status page — how probe targets are chosen (host-netns vantage), the three false-down shapes, triage… | +| [[Homepage Status Probes]] | How probe targets are chosen, the three false-down shapes, the triage procedure, and the e2e gate interactions. | | [[LAN-DNS]] | The homelab gives every service a name under `franksdank.cloud`. | | [[Loki]] | Loki is the log aggregation backend of the observability stack. | | [[Node-Exporter]] | Node-Exporter exposes host metrics (CPU, RAM, disk, network) to Prometheus. | -| [[Observability]] | The Dankclaw observability stack — Prometheus, Grafana, Loki, Alloy, node-exporter and Alertmanager (Signal… | -| [[Observability Provisioning]] | How Prometheus config/rules and Grafana datasources/dashboards are provisioned as code in serverAppConfigs — the… | +| [[Observability]] | The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured. | +| [[Observability Provisioning]] | Monitoring config as code: the config triad, deploy copiers, planner mappings, and where each change type lands. | | [[Pi-hole]] | Pi-hole is the LAN DNS server for the homelab. | -| [[Prometheus]] | Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints, and feeds the… | +| [[Prometheus]] | Scrapes each service's metrics endpoint and feeds the provisioned Grafana datasource; down targets are structural. | ## Repo architecture and policy | Page | Owns | |---|---| -| [[Container Network Namespaces]] | Which local services are addressable from which Podman network namespace on | -| [[Deploy State Triage]] | Reading and reconciling the deploy-state markers — stale marker vs live container, pending-service lockout,… | +| [[Container Network Namespaces]] | How Podman builds container networks here: the network inventory, the netns sharing mechanism, and the isolation limits. | +| [[Deploy State Triage]] | Reading deploy-state markers against live containers and deciding, safely, whether to redeploy. | | [[Deployment Flow]] | Service-scoped Git deployment for the two homelab hosts. | -| [[E2E Suite]] | `tests/e2e/` drives the **live fleet** through a real browser: fresh contexts | +| [[E2E Suite]] | The Playwright suite that drives the live fleet through a real browser, its project set, and the FrankBook conditional. | | [[Network Architecture]] | One Caddy fronts both hosts. | -| [[Quadlet Lifecycle]] | The full Quadlet discipline on this fleet — ownership auditing, compose-to-Quadlet cutover, drift/crash-loop… | -| [[Stale Signal Audit]] | The audit battery for surfaces that rot without failing — unreachable probes, vacuous assertions, invalid checkers,… | +| [[Quadlet Lifecycle]] | The Quadlet discipline: ownership audits, compose cutover, drift and crash-loop triage, generator quirks, log retrieval. | +| [[Stale Signal Audit]] | The audit battery for surfaces that rot without failing: dead probes, vacuous assertions, gates that never run. | | [[Storage Policy]] | All persistent data uses **bind mounts**: a host directory mapped directly into the container. | | [[Testing Policy]] | This repository tests its configuration at defined seams. | @@ -137,36 +132,46 @@ that endpoint's page, not a page. | Page | Owns | |---|---| -| [[Container Host Reachability]] | The two podman/netavark reachability quirks on the fleet — containers cannot dial unpublished host ports (LAN IP or… | -| [[Fleet Audit and Posture]] | Three standing audits — monitoring completeness (what should exist vs what is merely down), vhost proxy health… | -| [[Fleet Ops Playbook]] | Incident-shaped fleet procedures — LLM endpoint down with an abandoned GPU holder, scratch vLLM boot windows… | -| [[Payload and Disk Reclaim]] | Standing procedure for disk pressure on a fleet host — measure all buckets first, classify against the runbook index,… | -| [[Remote Job Runner]] | Detached long-job execution on fleet hosts with sentinel files (job.sh) — why ssh tail-watchers fail structurally,… | -| [[SSH and Remote Access]] | Passwordless SSH bootstrap without sshpass/expect, remote execution against fish login shells, and diagnosing a… | +| [[Container Host Reachability]] | Two reachability quirks: containers cannot dial unpublished host ports, and netns membership decides reachability. | +| [[Fleet Audit and Posture]] | Three standing audits: monitoring completeness, vhost proxy health, and behavioural security posture. | +| [[Fleet Ops Playbook]] | Incident-shaped procedures: endpoint down with a GPU holder, stop-first bench windows, service iteration, waking a host. | +| [[Payload and Disk Reclaim]] | Disk pressure procedure: measure every bucket, classify, prove nothing references the payload, stage, purge on approval. | +| [[Remote Job Runner]] | Detached long jobs with sentinel files: why tail-watchers fail structurally, the script, its invariants, verification. | +| [[SSH and Remote Access]] | Passwordless SSH bootstrap without sshpass, remote execution against fish login shells, and frozen-GUI triage. | ## Agent / harness | Page | Owns | |---|---| -| [[Agent Docs and Edits]] | Two self-inflicted damage classes — structurally broken markdown after parallel agent edits, and botched hashline… | -| [[Command Center Dotfiles]] | The ~/.omp repo is itself the deployment — stow-equivalent packages, full-replica rsync transport, the pre-overwrite… | -| [[OMP Harness]] | The omp agent harness as configured on this fleet — compaction chain arithmetic, per-model sampler routing via… | +| [[Agent Docs and Edits]] | Two self-inflicted doc damage classes from parallel agent edits, with the audit checks and recovery order for each. | +| [[Command Center Dotfiles]] | How the dotfiles repo deploys: stow-equivalent packages, full-replica rsync, the pre-overwrite audit, transport gotchas. | +| [[OMP Harness]] | The omp harness as configured here: compaction chain arithmetic, per-model sampler routing, and the context-lean audits. | ## Research and methodology | Page | Owns | |---|---| -| [[AI Job Search and Resume Tools]] | Self-hosted AI job-search and resume-formatting applications: the 2026 landscape, local-LLM fit, and a recommended… | +| [[AI Job Search and Resume Tools]] | Self-hosted job-search and resume tools: the landscape, local-LLM fit, and the recommended homelab stack. | | [[Backblaze Integration]] | Off-site backup plan for the bind-mounted data on both hosts. | -| [[ComfyUI ROCm and Vulkan]] | Research into whether ComfyUI can use GPU acceleration — not just CPU — on the AMD Strix Halo APU (gfx1151) in… | +| [[ComfyUI ROCm and Vulkan]] | Whether ComfyUI can use the Strix Halo APU at all, and which backend path actually reaches the GPU. | | [[Lemonade and sd.cpp]] | Why Lemonade Server can do image generation on Strix Halo when ComfyUI cannot. | -| [[Llama Cpp Code Diff]] | Code-level diff of mainline llama.cpp, charlie12345/ROCmFPX, and ik_llama.cpp with a focus on quantization code, ggml… | -| [[Llama Cpp Runtimes]] | Compared mainline llama.cpp, charlie12345/ROCmFPX, and ik_llama.cpp — positioning, quantization formats, code-level… | -| [[Local TTS and Voice Cloning]] | How to create a new local TTS voice from sample files in oh-my-pi, the 2026 open-weight voice-cloning landscape, the… | -| [[Most Popular Self-Hosted AI Apps]] | The 15 most popular self-hosted AI applications of June–September 2026, ranked by GitHub stars, with homelab fit and… | -| [[PICO-8 Asset Standards]] | Production-grade standards for PICO-8 asset sheets — palette discipline, single perspective and light direction,… | -| [[Service Research Brief]] | Consolidated research from the `frank-zealot` research subagents (2026-08-14) plus the pi agent knowledge base. | -| [[SGLang]] | What SGLang is, how it works (RadixAttention, paged KV, overlap scheduler, structured output), how it compares to… | +| [[Llama Cpp Code Diff]] | Code-level diff of the three llama.cpp trees here: quantization code, type numbers, and what differs for our cards. | +| [[Llama Cpp Runtimes]] | The llama.cpp trees on this host compared: positioning, quantization formats, and fit for each machine. | +| [[Local TTS and Voice Cloning]] | Creating a local voice from sample files: the open-weight cloning landscape, the measured engine A/B, the audio stack. | +| [[Most Popular Self-Hosted AI Apps]] | The most-starred self-hosted AI applications of the year so far, with homelab fit and adoption notes. | +| [[PICO-8 Asset Standards]] | Asset-sheet standards for PICO-8: palette discipline, one light direction, detail hierarchy, cohesion checklist. | +| [[Service Research Brief]] | Consolidated upstream research on the fleet's LLM runtimes, dated to when it was gathered. | +| [[SGLang]] | What SGLang is, how it schedules and caches, how it compares to the runtimes here, and whether it fits this fleet. | + +## Pipeline Parallelism (series) + +| Page | Owns | +|---|---| +| [[Ethernet Bandwidth Math]] | Bytes-per-token math for pipeline versus tensor parallelism, and the link budgets that follow from it. | +| [[Expected Performance]] | Estimated split-model throughput over ethernet, with the benchmark anchors the estimates rest on. | +| [[Multi-Node Runtimes]] | How each two-machine runtime is built and launched, what it can combine with, and the mixed-vendor verdicts. | +| [[Qwen3.8 PP Deployment]] | The real split deployment: which node is head and which is worker, the flags, and the verified gotchas. | +| [[Sample Script Set]] | An educational two-node RPC split script set, for reading only — never run it or deploy it. | ## LLM technique and hardware references @@ -174,62 +179,51 @@ that endpoint's page, not a page. |---|---| | [[Dankclaw Improvements]] | This page collects concrete, sourced recommendations to improve Dankclaw | | [[Dankclaw System Specs]] | Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. | -| [[dankclaw-iommu-translation-off]] | Dankclaw boots iommu=pt since 2026-10-18 — identity DMA with the IOMMU armed, which restored the XDNA2 NPU at a… | -| [[Dankserver System Specs]] | Dankserver hardware and OS state since the 2026-09-20 board swap (ASUS ProArt B850-Creator WiFi NEO, 2x AMD Radeon AI… | -| [[dankserver-frontier-moe-feasibility]] | VERIFIED 2026-09-28 by measurement and source reading. Compares DeepSeek-V4.1-Flash, MiMo-V2.6-Flash-MOPD and… | +| [[dankclaw-iommu-translation-off]] | Why Dankclaw boots with identity DMA, what the earlier off-era bought and cost, and which tooling owns the boot line. | +| [[Dankserver System Specs]] | Dankserver hardware and OS state since the board swap: board, lane topology, cards, memory, kernel posture. | +| [[dankserver-frontier-moe-feasibility]] | The capacity arithmetic that decides whether a frontier mixture-of-experts fits this box before any model is loaded. | | [[dankserver-vram-growth]] | Investigation of the recurring cuBLAS crash on frank-zealot. What grows, what does not, and what to do next. | -| [[dealign-ornith-crack]] | dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with… | | [[deployment]] | Scoped deployment and update process for the three inference services (frank-archon, frank-dragoon, frank-zealot). | | [[DFlash DSpark Speculative Decoding]] | DFlash and DSpark are the two newest speculative decoding methods. | -| [[Dual-R9700-Cooperation]] | Why frank-medium's two Radeon AI PRO R9700s could not run tensor-parallel on the old B650-E board (PCIe atomic… | -| [[FastFlowLM NPU Inference]] | FastFlowLM (FLM), the AMD NPU-first LLM runtime for Ryzen AI XDNA2 chips. Fit analysis for the 50 TOPS NPU on… | -| [[FreeToken Builds]] | Building FreeToken (the official franksdank fork of FreeTok Tokenworks' freetoken vLLM-MTP runtime) on Dankclaw — the… | -| [[freetoken-benchmark-sweep]] | KV pool vs decode speed sweep across bs1/bs3/bs4/bs6 on Dankserver with Qwen3.6-35B-A3B-FP8 via FreeToken v0.1.2. | -| [[GGUF Compatibility]] | How to answer "will engine X load this GGUF?" from the file's header alone, why a custom tensor type forces the… | -| [[Heterogeneous Split Triage]] | Triage before building when someone asks to split a model across unequal accelerators on one host (24 GB GDDR6X +… | +| [[Dual-R9700-Cooperation]] | Why two R9700s could not run tensor-parallel on the old board, and what the board swap unblocked. | +| [[FastFlowLM NPU Inference]] | The AMD NPU-first runtime: fit analysis for the XDNA2, published benchmarks, install state, and test runbook. | +| [[frank-medium-batch-ubatch]] | The batch-versus-ubatch sweep on the layer-split pair at its deployed shape, and the rebalance the sweep forced. | +| [[FreeToken Builds]] | Building the FreeToken runtime here: the CUDA and ROCm builds and the dependency trap each one hit. | +| [[freetoken-benchmark-sweep]] | KV pool versus decode speed for the FP8 mixture-of-experts payload on the FreeToken runtime. | +| [[GGUF Compatibility]] | Answering whether an engine loads a GGUF from the file header alone, and pinning a fork build that can. | +| [[Heterogeneous Split Triage]] | Triage before splitting across unequal accelerators: link characterisation, real bottleneck, what placement never buys. | | [[Hybrid CPU GPU Processing Dankserver]] | Dankserver uses NVIDIA GPU with system RAM for hybrid processing. | -| [[ling-3.0-flash-trial]] | `bailingmoe3-support` (commit `db4480bc8`, build b10411) at `/home/dank/llama.cpp-ling`. | -| [[ling-3.0-flash-vllm-plan]] | This page was the plan for a vLLM-ROCm deployment of `jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native` (INT4 W4A16, ~62… | -| [[Llama Cpp Multi-Device Splits]] | What llama.cpp can and cannot express across devices, the no-collective layer-split path, split balancing by… | -| [[Llama Cpp Serving Sizing]] | Slot-pool sizing against a headroom band, per-device VRAM accounting, prompt-cache/state sizing and the QSA… | -| [[LLM Bench Windows and Measurement]] | Stopping-and-restoring benchmark windows on the fleet LLM hosts (llama-bench sweeps, HTTP param benches, multi-arm… | -| [[LLM Performance Diagnosis]] | Probing why a served LLM is slow — the flat-depth probe, the expert-offload duty-cycle arithmetic, kernel-time… | -| [[Model Swap and Rollback]] | The full model-swap discipline for the LLM fleet — fit preflight, fetch and payload verification, runtime build/flag… | -| [[PCIe Link and Atomics]] | Link probing (idle downshift, path ceilings, CUDA/HIP bandwidth probes), the streaming-vs-compute ceiling arithmetic,… | +| [[ling-3.0-flash-trial]] | Trial of a fork build for a new mixture-of-experts architecture: what loaded, what the harness refused, and the verdict. | +| [[Llama Cpp Multi-Device Splits]] | What llama.cpp can express across devices: the no-collective layer-split path, balancing, and drafter KV sizing. | +| [[Llama Cpp Serving Sizing]] | Measurement-first sizing for llama.cpp shapes: slot pools against a headroom band, per-device accounting, prompt cache. | +| [[LLM Bench Windows and Measurement]] | Opening and restoring benchmark windows on fleet hosts, honest methodology, and the usage-counter law under drafting. | +| [[LLM Performance Diagnosis]] | Probing why a served model is slow: the flat-depth probe, offload duty-cycle arithmetic, and profiler attribution. | +| [[Model Swap and Rollback]] | The model-swap discipline: fit preflight, fetch verification, build and flag checks, live validation, three-pin revert. | +| [[PCIe Link and Atomics]] | Link probing, the streaming-versus-compute ceiling arithmetic, and the atomics wall that decides multi-GPU collectives. | | [[Quantization by Hardware Architecture]] | This page maps LLM quantization formats to the GPU and CPU architectures that support them. | -| [[Quantization Production]] | — | -| [[qwen3.8-strix-optimizations]] | This page answers one question: **how do we make Qwen3.8-27B serve faster on | +| [[Quantization Production]] | Making a quantized payload here: the conversion recipe, the scoring gate, and the campaign ladder behind a shipped file. | +| [[qwen3.8-spec-matrix]] | The speculative-decoding benchmark history for this endpoint, arm by arm, and what each arm ruled out. | +| [[qwen3.8-strix-optimizations]] | How a served checkpoint got faster on the Strix Halo iGPU: the lever ladder and what each lever actually bought. | | [[RAM CPU Offloading in VLLM]] | vLLM uses CPU RAM and swap space to extend GPU memory capacity. | -| [[RDNA4 Container Serving]] | Bringing a model larger than VRAM onto Dankserver's 2x R9700 (gfx1201) with the tcclaviger vLLM stack — quant-class… | -| [[RDNA4 MoE Expert Offload VRAM Budget]] | The expert-offload plan arithmetic on the tcclaviger fork — which terms a deploy can move, the util law, the… | -| [[Running Qwen3.6 27B FP8 on Dankclaw]] | Qwen3.6-27B is a dense vision-language model from the Qwen3.6 series. | -| [[Running Qwen3.6 35B A3B FP8 on Dankserver]] | Qwen3.6-35B-A3B is a Mixture-of-Experts vision-language model from the Qwen3.6 series. | +| [[RDNA4 Container Serving]] | Bringing a model larger than VRAM onto the two-card box: quant gating, fetch verification, container GPU-init triage. | +| [[RDNA4 MoE Expert Offload VRAM Budget]] | Expert-offload arithmetic on the RDNA4 fork: movable terms, the utilisation law, graph charge, ceilings, context ladder. | +| [[Running Qwen3.6 27B FP8 on Dankclaw]] | Serving a dense FP8 vision-language checkpoint on the Strix Halo iGPU: the load path, the flags, the format gate. | +| [[Running Qwen3.6 35B A3B FP8 on Dankserver]] | Serving a mixture-of-experts FP8 checkpoint on the R9700 host: the load path, the format gate, the offload shape. | | [[Shared Memory Architecture Dankclaw]] | Dankclaw uses AMD Strix Halo APU with unified memory architecture. | -| [[swift15-campaign-worklog]] | — | -| [[Tcclaviger Fork Ops]] | Operating the tcclaviger RDNA4 vLLM fork on dankserver (shape G era, rollback target since the 2026-10-05 FreeToken… | -| [[ubatch-benchmark-large]] | Standardized ubatch sweep for the active kingjones Qwen3.8 ROCmFPX deployment. | -| [[vLLM Boot Forensics and Config Interrogation]] | Capturing a vLLM container boot failure's real root cause from journald (the container log is already gone),… | -| [[VLLM Index]] | This index contains all documentation for vLLM deployment on Dankserver and Dankclaw. | +| [[swift15-campaign-worklog]] | Dated worklog of the campaign that converted and served a Swift checkpoint on the radiance endpoint, arm by arm. | +| [[Tcclaviger Fork Ops]] | Operating the RDNA4 vLLM fork: the sweep harness, measured knob verdicts, sweep hygiene, compile-cache persistence. | +| [[ubatch-benchmark-large]] | The wide ubatch sweep for a deployed ROCmFPX shape, run through the standardized harness. | +| [[vLLM Boot Forensics and Config Interrogation]] | Recovering a container boot failure's real cause from journald, and interrogating a live endpoint's true config. | +| [[VLLM Index]] | Locator for every vLLM page in this vault. | | [[VLLM Overview]] | vLLM is a fast and easy library for large language model inference and serving. | -| [[vLLM Porting and Compile Triage]] | Forward-porting a model architecture to a new vLLM major (additive-only discipline, gap enumeration, shims that… | -| [[vllm_research_findings]] | vLLM v0.26.0 (latest stable, released July 25, 2026) supports DFlash speculative decoding with the `--spec-method… | -| [[x4-pair-experiment]] | What mainline llama.cpp can and cannot do with a 3090 + Strix Halo combination, measured 2026-09-16 on Dankclaw — the… | +| [[vLLM Porting and Compile Triage]] | Forward-porting a model architecture: additive-only discipline, shims that survive worker spawn, compile triage. | +| [[vllm_research_findings]] | Upstream support survey for speculative decoding: which flags, releases and pull requests carry each method. | +| [[x4-pair-experiment]] | Measured limits of a consumer card plus APU pair: the tensor inventory, the flag that kills cross-device placement. | -## Pipeline Parallelism (series) +## Change history | Page | Owns | |---|---| -| [[Ethernet Bandwidth Math]] | Bytes-per-token math for pipeline vs tensor parallelism, feasibility tables for 1/2.5/10 GbE, and real multi-node… | -| [[Expected Performance]] | Estimated tokens per second for a 25-40B model split across two nodes over 1/2.5/10 GbE, with real benchmark anchors. | -| [[llama.cpp RPC]] | llama.cpp distributed inference via ggml-rpc-server and the --rpc flag: architecture, CLI, versions, limitations, and… | -| [[Mixed-Vendor Experiments]] | Real-world NVIDIA + AMD Strix Halo multi-node experiments: what works (llama.cpp RPC, Vulkan split), what breaks… | -| [[Qwen3.6 27B Spec Methods]] | Qwen3.6-27B Q8 GGUF variants and llama.cpp speculative decoding (draft-mtp, draft-dflash, ngram-mod): flags, RPC… | -| [[Qwen3.8 PP Deployment]] | Split Qwen3.8-27B across the Strix Halo iGPU (head) and the RTX | -| [[Sample Script Set]] | An educational two-node llama.cpp RPC split of Qwen3.6-27B-Q8_K_P over ethernet. For reading only. Do not run or… | -| [[vLLM Pipeline Parallelism]] | vLLM multi-node pipeline parallelism: flags, Ray/mp backends, interconnect requirements, PP bubble, and PP +… | +| [[log]] | Chronological incident and law log for the whole bundle. | -## Uncategorised - -| Page | Owns | -|---|---| -| [[log]] | — | + diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 604b030..0a8d603 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -1,5 +1,52 @@ # Bundle Change Log +## 2026-10-19 +- **Thirteen vault pages were the same page twice, and the doubling was a + frontmatter tool, not a human.** `frank-archon`, `frank-dragoon`, + `frank-zealot`, `frank-npu`, `frank-redo-2026-08-17`, `kingjones-q8agent-rocmfpx` + and the five `frank-medium-*` era pages measured exactly 2× their `HEAD` size, + each with a second frontmatter block whose closing `---` was glued onto the end + of a body or timestamp line. **Law: a rewrite tool that re-reads the file it + just wrote reads its own output as the body, and a fence-repair pass keyed on + `line == "---"` never sees a glued fence** — the repair then inserts a fence in + the middle of the page and the next pass treats the tail as a new document. + Un-double rule that worked: cut where a `type:` line follows the first + frontmatter's close with more keys behind it, keep the copy whose body is a + superset, and prove it with the orphan census (`0` losses) before writing. +- **`.AGENTTRASH` is not vault content, and three tools forgot.** The link + checker resolved links *into* the trash (a live page pointing at a deleted page + read as healthy) and the docs seam demanded an index row for every trashed page. + Both now skip it, `check-okf.py` and `gen-vault-index.py` already did. +- **A cancelled agent's report is not evidence.** One subagent was killed by the + task runtime limit and its yield still described a 495-line owner page, a + verified era table and a byte-level census — the file did not exist, and the + table it quoted was self-referential. Another wrote its output to disk and then + died without reporting, leaving a compressed `log.md`. Both were caught only by + `wc`/`ls` on the file itself. **Law: verify the bytes, never the narrative** — + re-derive existence, line count and the orphan census after any delegated write, + whatever the yield claims. +- **`log.md` structure per OKF: one entry per date, bare `## YYYY-MM-DD`, newest + first.** 66 dated headings collapsed to 34 by merging same-date entries and + demoting their titles to bold leads; the file had been descending at the head + and ascending at the tail. Kept every line — an attempted prose compression was + rejected because the census showed it dropped laws held nowhere else (the + `qsa_sparse` 64-page-size override that refused 300,000 tokens, the + `{"status":"loading"}` health response that false-greened a deploy at six + seconds, the `libhiprtc.so` dev symlink, the stale `~/.docker/config.json` that + poisons podman's anonymous pulls). +- **Two date claims about the vllm-radiance era on the Dankserver port do not + reconcile, and neither was overwritten.** [[frank-archon]]'s Eras table opens on + 2026-09-26; [[frank-medium-radiance-mxfp4]] dates its own retired era to + 2026-10-10; nothing in the repo covers 09-26 → 10-10 for port 7770. The new + [[frank-medium]] locator records the dispute instead of picking a winner. +- **Host census settles a rollback claim.** `/home/dank/models/` on dankserver + holds `StillDeadcode`, `rad`, `tpl.jinja` — so the MXFP4 checkpoint that two + pages called the "on-disk rollback" is a re-fetch, now stated once, on the era + page that owns that payload. +- **Fleet state confirmed live, not from files**: frank-archon 3 × 200,000 fp8 KV, + frank-dragoon 2 slots × 200,704, frank-zealot 2 × 262,144; all three answer + `/health` 200 and every env value matches its page. + ## 2026-10-18 - **frank-zealot resliced to two windows of exact KV (Frank's call, PR #21).** `--ctx-size 786432 --parallel 3` at q8_0 K+V → **`524288 / 2` at f16 K+V**, so @@ -24,6 +71,729 @@ `/v1/chat/completions`. `AGENTS.md` said `/completion`; it now says the chat path. +**issue #3 research: AMD PerfOpt retires the reason for `amd_iommu=off`** + + +- Ask: does a recent kernel change remove the IOMMU translation penalty + that justifies the 2026-09-11 `amd_iommu=off` boot line, i.e. can the + XDNA2 NPU come back? Answered and documented; **no host change** at the + time — the fix is not installable from the Ubuntu 26.04 archive. +- The change is **AMD PerfOpt** (spec §3.4.9, MMIO `0x016C` bit 13): merged + via the iommu tree 2026-09-24, first release **Linux 7.4**. It stops + GPA→SPA translation for the *privileged iGPU* only and leaves the IOMMU + fully armed for every other device, NPU included. Its only consumer + requires `AMD_IS_APU` **and the identity domain** — so PerfOpt rides on + top of `iommu=pt`, it does not replace it. +- **Three premises of the issue corrected.** (1) The 13–16 % is the + *vendor's* A/B against *their* previous boot line — our record copied the + figure on 2026-09-11 without ever measuring here. (2) The NPU does **not** + have to wait for 7.4: `iommu=pt` enables the IOMMU, which is all + `err=-19` asks for (`amdxdna` binds via `iommu_sva_bind_device()` into its + own SVA domain, independent of the default domain type). (3) The 460 + tok/s baseline is stale, so the window must measure `off` vs `pt` locally + and inherit no number. +- Silicon check: our IOMMU is `1022:1508` Strix/Strix Halo rev 02, and the + **Strix Halo desktop** arm measured only **2–4 %** from PerfOpt (18–23 % + on Strix Point / Krackan laptops) — the modest arm is the one that + describes this box. Kernel pick: **skip 6.18–6.18.7**, where an IOMMU + regression disables SVA outright. +- Self-correction inside the same session: the first pass claimed + `iommu=pt` alone already avoids the cost. It does not — an identity + attach keeps the page walk and enables ATS/PRI/PASID/GCR3, so `pt` + recovers **most** of the gap and PerfOpt is the last part of it. The + posted comment (Gitea issue #3) was edited to the corrected text. +- Artifacts: a new section in `dankclaw-iommu-translation-off.md` + (front-matter re-keyed to "vendor-claimed, never measured locally"), + `frank-npu.md` § *Why it is retired* corrected — the "`pt` is not a + middle ground" law is dead — the AGENTS.md key-lesson bullet re-keyed, and + a 4-step maintenance-window procedure that measures locally instead of + inheriting the vendor number. Nothing deployed — no service was touched. + +**boot line flipped: Dankclaw now boots `iommu=pt`** + + +- Window executed 01:24–01:35 on Frank's go-ahead, with the A/B the + research demanded. Provenance first: `/etc/default/grub` had never been in + the repo while the IOMMU line was the most consequential kernel decision + on the box, so it is now tracked at `dankclaw/host/grub` (fetched + byte-exact) and moves only through `scripts/set-dankclaw-bootline.sh` — + diff, install 0644 root:root, `update-grub`, **verify the default + `grub.cfg` entry**, reboot on `--reboot` only. The deployer plans nothing + under `dankclaw/host/`: kernel state is not a container restart. +- **Verdict: identity DMA costs nothing on this host.** Byte-identical + traffic, +2.0 % cold prefill going OFF→PT; the 13–16 % that justified + `amd_iommu=off` for five weeks never existed here. Decode's apparent + −9.7 % was instrument noise: three repeats of the same request on the + same boot spread 31.8–47.8 t/s. +- **The XDNA2 NPU is back:** `amdxdna` bound `0000:c6:00.1`, + `/dev/accel/accel0` `crw-rw---- root:render 261,0` with `dank` in + `render`. No endpoint deployed — revival needs a quadlet with + `AddDevice=/dev/accel/accel0` + `GroupAdd=keep-groups`. +- Deliberately NOT done: the `pp_dpm_fclk` level-7 pin (it changes iGPU + timing for every workload and nothing co-runs on the NPU yet) and any + PerfOpt work (kernel `7.0.0-38` has zero `perfopt` symbols; the 26.04 + archive caps at `7.0.0-38.38`). +- Gates after boot: 0 failed units, 32 IOMMU groups (31 identity + 1 DMA = + the Genesys Logic SD reader, a kernel quirk), interrupt remapping on; + frank-dragoon's ledger byte-for-byte the same shape and Gitea / Caddy + edge / Grafana answered 200. New tool `scripts/stage-frank-dragoon-eval.sh` + rebuilds the A/B corpus byte-identically — run it through `bash -lc`, `uv` + is in `~/.local/bin`. Docs re-keyed: the decision record + `dankclaw-iommu-translation-off.md` (new § *execution*), `frank-npu.md` + (precondition satisfied, endpoint still absent), AGENTS.md key lesson. + +**frank-zealot: Bonsai-2-9B PQ2_0 with the MTP head (3 × 262,144)** + +- 2026-10-18 — frank-zealot: 27B PQ2_0 → **Ternary-Bonsai-2-9B PQ2_0 with + the MTP head**, runtime moved to the PrismML vendor fork build **10777** + because the MTP arm needs the build that pairs MTP with a quantized V + cache. See [[frank-zealot]]. +- The win is the **window, not the quant**: every conversation goes back at + the trained 262,144 (the 27B era sold each slot 200,704), and the fleet's + widest context makes :7771 the **compaction hub** — `models.yml` + `contextWindow` 200704 → 262144. +- **MTP armed, then OFF by measurement.** The head is in the file and the + fork loads it, but the draft context bills its own **f16** KV over the + WHOLE pool, and at `UBATCH_SIZE=2048` that squeeze **halved decode**. The + documented MTP shape (`SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024`) was + declined — the decode gain is not worth spending the free card on it. The + arm is recorded in the env header, not left to rediscovery. +- Two bugs the trial caught, both fixed in the repo: + `${SPEC_PRESENCE_PENALTY:=0.0}` was a *default-assignment*, so the env + value always won and the documented "MTP runs without + `--presence-penalty`" arm was unreachable — now `${VAR=0.0}`, which lets + `SPEC_TYPE=draft-mtp` drop the flag. And `up{job="frank-zealot"}` had read + 0 for the **entire retained Prometheus history**: llama.cpp answers + **501** on `/metrics` unless `--metrics` is passed, so the scrape failed + beside a healthy `/health` and `LlmEndpointDown` fired for days. + `METRICS=1` is now the default and the seam pins it. +- Deployed 13:16 (`deploy.sh dankclaw --service frank-zealot`) — and the + first boot was **not the shipped payload**: a leftover + `frank-zealot.service.d/ptq1.conf` from the bench window pointed + `FRANK_ZEALOT_ENV_FILE` at `/tmp/zealot-arm-current.env`, so the unit + booted that arm. `/health` 200, `/v1/models` echoed the alias, Playwright + 33/33 — **nothing in the gate reads the payload.** Drop-in moved to + `~/.AGENTTRASH/`, `daemon-reload`, restart: the shipped trio booted. (The + drop-in was not a rogue — see the PTQ1_0 entry below.) +- Measurement law: the first battery reported 117 and 72 t/s because it + timed natural generations that stopped at 24 tokens — wall-clock aggregate + on an early EOS reads 4× low. `ignore_eos` + the engine's own `timings` + numbers is the only honest way to time this endpoint. +- OpenWebUI `model` rows re-keyed in the live DB. Note for the next edit: + the values live inside the `params`/`meta` JSON columns — the table has no + `context_length`/`description` columns — and there is no `sqlite3` binary + on the host, so `python3` stdlib + `Connection.backup()`. + +**frank-zealot PTQ1_0 arm: measured, approved, and never written down** + +- **Correction to the entry above and to my own report.** The + `frank-zealot.service.d/ptq1.conf` drop-in was NOT a half-finished + experiment. The session archive (`01a14c93`) carries Frank's own + instruction at **02:36**: replace frank-zealot with + `Ternary-Bonsai-2-27B-PTQ1_0.gguf`, use the **Q8** mmproj, batch/ubatch + **3072/1536**, "fit as much context as possible in 2 contexts". +- It was the production payload **03:14:39 → 03:38:57** and served real + traffic; recovered from its own `print_timing` lines, PTQ1_0 ran + ≈47.6–49.0 t/s single-stream greedy decode (25–27 t/s per stream with two + streams live) and prefilled deep prompts faster than the PQ2_0 27B record + ever managed. +- Against the PQ2_0 27B record: PTQ1_0 freed the **1,201 MiB** its fetch + script promised and paid **−37 % decode** — the publisher's "dense + ternary unpacking loses on Ampere" warning, now confirmed on GA102. + Frank scratched the whole 27B plan at **03:33** ("scratch the current + frank-zealot plan completely… 9B… enable mtp… if you can possibly fit a + third context, do it"), which is what shipped at 13:16. +- The durable failure is mine: those numbers lived only in `/tmp` and the + journal, never in the repo, so the next agent on the host could not tell + an approved payload from a rogue drop-in and called it a hijack. The + fetch script's header now carries the measurement; + `/home/dank/models/Bonsai2-PTQ1_0/` stays on disk as the 27B rollback and + the drop-in stays trashed. + +**frank-dragoon: the 401,408-position pool goes live; docs re-keyed to the measured boot** + + +- `./scripts/deploy.sh dankclaw --service frank-dragoon` — e2e 33/33, and + the drop-in check the zealot incident bought ran FIRST: `DropInPaths` + empty, so the repo env governed. 2026-10-18 — frank-dragoon: the KV pool + doubled to 401,408 positions over the 200,704 ceiling; the measured boot + ledger owns the shape. See [[frank-dragoon]]. +- Warm reuse survived the raise: one **101,987**-token prompt took 81.5 s + cold and 0.3 s on repeats — `HALOGEN_CACHE_INPLACE=1` keeps the KV rows in + the pool, so a warm turn prefills nothing and the doubling evicted nothing + the cache needed. The RAM price is still real: host `MemAvailable` fell + **26 → 22 GiB**, because every pool position is RAM the n-gram table's + page cache loses. +- Every document quoted **~10.8 GiB**, the ~28 KiB/position arithmetic; the + engine reserves **11.0 GiB** and its preflight bills **11.3**. The measured + figure wins, and every "committed but not yet deployed" clause was retired + with the measurement (AGENTS.md, the env lore block, both index pages, the + iommu page, the runbook). +- PTQ1_0 provenance corrected (see the dedicated entry): the arm that + hijacked the zealot deploy was a **repo-tracked, Frank-ordered, measured** + candidate — `scripts/download-frank-zealot-bonsai2-ptq1.sh` is committed, + and its header now carries the recovered numbers (≈48 t/s decode, 801 t/s + prefill at 141k) and the verdict: −37 % decode against PQ2_0 for 1,201 + MiB freed, then scratched by Frank's 03:33 order in favour of the 9B. + Nothing is owed on it. +- Census of the halogen HTTP surface while doing it, because the fleet bench + law quoted the wrong engine: frank-dragoon answers `/health`, `/metrics`, + `/v1/models`, `/v1/chat/completions`(POST) and 404s `/ping`, `/props`, + `/completion`, `/tokenize`; the chat response carries **no + `Server-Timing`** header. So the `/completion` `timings` object and the + `ttft;dur=` header are **llama.cpp** instruments — frank-zealot's, not + dragoon's. AGENTS.md's Benchmarking Policy re-keyed to say which engine + owns which instrument. + +**frank-tts phase 1: the cloned-voice endpoint, named and tested (not deployed)** + +- Voice-over goal, phase 1 (Gitea issue #2). New service **frank-tts** on + Dankclaw: `dankclaw/voiceservice/` (stdlib-only `app.py` + `Containerfile` + + rootless quadlet), `scripts/frank-tts.env`, `tests/frank_tts_unit.bats`. + One OpenAI-shaped `POST /v1/audio/speech` surface; `voice=` is + honoured only when `/voices//reference.wav` exists, otherwise a 400 + listing the voices that are — **no silent fallback voice**; oversized + input is a 414, not a half-narrated sentence; `/health` reports backend + + voice list; no audio ever reaches the log. +- **Naming law applied (Frank's call):** treat it as a first-class model + endpoint → alias-named `frank-tts` (TTS endpoint, so not `frank-asr`), + **port 7772**, joining `7770 frank-dragoon` / `7771 frank-zealot`; the + engine upstream keeps 8880 and is deliberately not a fleet endpoint. The + internal knob names stay `SHIM_*` (they name the façade component, not the + service). +- **Voice-pack law recorded** in the env header: a pack needs + `reference.wav` (24 kHz mono, 18–25 s) **and** the pinned engine's own + speaker artifact — required, beside the WAV, never replacing it — plus + `manifest.json` provenance whose offsets are always past the first 10 s of + the source (the source-material law). Enforcement is split honestly: the + shim gates on `reference.wav` (the only engine-agnostic term); the pack + builder is what will refuse to emit a pack without its artifact. +- Landed as **PR #4** (branch `wt/20261017-181502` → `main`, merged as + `49c4e19`) — the first PR on this repo through the API, which cost three + 422s to get right: the merge endpoint takes **`{"Do":"merge"}`**, `Do` + being a *string* naming the strategy — `DoMerge`/`merge_method` do not + exist on this Gitea, and swagger JSON is disabled on the instance, so the + 422 messages are the schema. Law saved to the `gitea-api-pr-operations` + skill. + +**TTS engine picked: arm A `qwentts.cpp`, on the live-output criterion** + +- Frank listened to the four audition pairs (`qwentts-*.mp3` against their + `armB-*.mp3` mainline partners) and **passed every clone**. That deleted + the only axis the A/B could not measure, and the question was reframed: + **which engine is better for live output**. On that criterion RTF only has + to stay under 1.0 with margin — the discriminators are time-to-first-audio + and run-to-run spread, because an arm whose RTF can cross 1.0 stalls a + live stream mid-sentence. +- **Chosen: arm A `qwentts.cpp` @ `5916bf7`, the resident `tts-server`, + 1.7 B talker Q4_K_M + Q8_0 tokenizer, in a `frank-tts-engine` container on + 8880.** The `TTS_BACKEND=llama-tts-cli` path in `scripts/frank-tts.env` + stays in the repo as the rollback. Three consequences are configuration, + not preference: boot-warm every voice, do not pin a cpuset (threads are + `hardware_concurrency()/2`), register every pack with its WAV (a lone + embedding is rejected). +- Two follow-ups deliberately **not** decided: the **0.6 B talker** (never + auditioned — every clip is 1.7 B) and **ICL vs embedding-only** + conditioning. Embedding-only is the shipped default because it is cheaper + and needs no `ref_text`, which this fleet cannot transcribe honestly; + `.spk` + `.rvq` still ride in every pack, so ICL remains a per-voice + opt-in for batch work where total time beats first byte. +- **Addressing law corrected** in [[Local TTS and Voice Cloning]]: the pair + may not meet at a host-loopback publish. netavark DNATs published ports + only and a bridge container cannot reach a host `127.0.0.1` bind — the + lesson that 502'd hermes — so `host.containers.internal:8880`, which + `scripts/frank-tts.env` documents, is correct only for a routable bind. + The deployed pair shares one quadlet network and the façade addresses the + engine by container name. +- **PR #12** (`feat/open-notebook-podcast` @ `114dd7a`) is **merged** as + `1eb8b21` (Frank's OK): open-notebook + SurrealDB quadlets on the private + bridge `10.89.4.0/24`, the API published on **5056** because seerr holds + `127.0.0.1:5055`, and `ESPERANTO_TTS_TIMEOUT` moved 600 → **660** to sit + one round-trip outside frank-tts's own `SHIM_TIMEOUT`. Merged is not + runnable: nothing listens on 7772 until the voice stack lands, and the + episode destination stays interim pending Gitea #7. + +**open-notebook: the podcast generator's config lands (nothing deployed)** + + +- Goal: **open-notebook** as the fleet's podcast generator on Dankclaw — + notes and sources in, a multi-speaker episode out, script written by + **frank-zealot**, voices rendered by **frank-tts**, no cloud provider + anywhere in the path. Repo-side only: no host touched. Runbook: + [[Open Notebook]]. +- **Two containers, not one.** `lfnovo/open_notebook:v1-latest` plus + `surrealdb/surrealdb:v2`, both **digest-pinned** with `Pull=never` — + frank-dragoon's pinning style, because `:latest` moving underneath a + rootless service is how you get an unbootable pod after an unrelated + `systemctl-daemon-reload`. Both health probes are argument-only + (`curl -fsS …/health`, `/surreal is-ready`), and `is-ready` refuses to + pass without `SURREAL_PASS` in the environment — a credential that must + never ride the command line, so `Exec=` carries flags only. +- **Host 5055 was already spoken for** — seerr has held `127.0.0.1:5055` — + so the API publishes `0.0.0.0:5056:5055`, and because open-notebook's + frontend derives its API URL from the window origin when `API_URL` is + unset, `API_URL=http://192.168.0.143:5056` is set to match: a silent + remap would hang the UI on a port nobody listens on. 8502 publishes on + `0.0.0.0` because this service carries its own password (the + episode-audio route is unauthenticated); **8000 is published nowhere**, + where upstream exposes it on loopback with root credentials, and the seam + forbids it. +- **The CPU guard, pre-armed.** `frank-tts` spawns one ~8 GiB, 8-thread + engine per request on a box that also serves two LLMs, while + open-notebook defaults to a 25-way self-DoS; both knobs are pinned to + `1` in `scripts/open-notebook.env`. The TTS clock goes to **660 — one + round-trip outside** frank-tts's own `SHIM_TIMEOUT=600`, so a shim + give-up reads as the shim's error rather than the worker's; the seam + asserts the direction (`>`) against the frank-tts env instead of + restating a number. +- **What a credential needs in order to exist:** + `OPEN_NOTEBOOK_ENCRYPTION_KEY` must be set before the UI can save *any* + provider credential, and `OPEN_NOTEBOOK_PASSWORD` must stay **absent** to + remain auth-off — the app reads `os.environ.get(…) is not None`, so + `PASSWORD=` boots a login form asking for the empty string. The deployer + seeds `/home/dank/Containers/open-notebook/env` (600) from the host vault + and **refuses to start the pair while either secret is blank**, because a + half-configured open-notebook stores no credentials and fails quietly. +- **Episode destination is interim, by decision.** The destination is + **Gitea issue #7, `human-only`** — agents must not work it. Until Frank + settles it, the app keeps its own named volume and + `/home/dank/media/podcasts` is bind-mounted over `/app/data/podcasts` + (`DATA_FOLDER` is hard-coded `./data`), so finished MP3s are ordinary + files in his home directory instead of volume contents needing + `podman unshare`. One mount line; one deletion undoes it. +- **Two UI buttons will fail, and that is cosmetic:** Test Connection does + `GET /models` and the TTS probe synthesises voice `alloy`; + frank-tts serves `/health` and `/v1/audio/speech` only. Model rows are + created *by name* and the voice field is free text, so `narrator1/2/3` is + typed in and never discovered. Documented, not worked around. +- Three deployer seams cost time and are now law-shaped: a service must be + registered in **five** places — `SERVICE_ORDER`, `DANKCLAW_SERVICES`, the + `deploy-remote.sh` dispatch case, its pre-deploy requirement, and + `deploy.sh`'s dispatch regex, where an omission silently plans an empty + set for `--only`; a `dankclaw//*` case must precede the + `dankclaw/*.container` glob because that `*` crosses `/`; and the + changed-path routing lives *inside* the per-path loop, after the + `scripts/*.env)` case, not at the top of the file. + +**frank-tts-engine: the arm A engine becomes deployable (nothing deployed)** + +- Goal: turn the engine the day's A/B picked into a **repo-owned, + boot-persistent, self-healing container** — `frank-tts-engine`, + `qwentts.cpp` commit `5916bf7c79dbb4e66fc059cdc507a79a23a28bb4`, resident + `tts-server`, container port 8880, and **not** a fleet endpoint: the + endpoint stays `frank-tts` on 7772. Runbook: [[Frank TTS Engine]]. +- **No `PublishPort` at all, which the merged A/B states as law.** netavark + DNATs published ports only, so a `127.0.0.1:8880` publish is not a path to + a peer container (the trap that 502'd hermes) — while being the one + LAN-reachable surface of an unauthenticated synthesiser. Both voice + containers therefore sit on one committed quadlet network, + `frank-tts-engine.network` at **10.89.5.0/24** (the next free /24 after + open-notebook's 10.89.4.0/24), the façade addresses the engine by + container name, and diagnostics — including the deployer's health gate + (`wait_container_healthy`, there is no URL to dial) — run through + `podman exec`. +- **The boot warmer is a unit hook, not an operator habit.** The registry is + memory-only and the first request for a registered voice decodes the whole + reference, so `ExecStartPost=-/usr/bin/podman exec frank-tts-engine + /app/warm-voices.sh` with `TimeoutStartSec=900`; the `-` prefix is what + stops a failed warm from crash-looping the endpoint under + `Restart=always`, and it logs `cold=` and `warm=` per voice on every boot. +- **Thread law recorded before anyone tunes it away:** no `--threads` + exists; `src/backend.h` uses `hardware_concurrency()/2`, so a cpuset pin + silently changes the thread count. Nothing pins this container. +- Image builds from the pinned commit with the **proven** line, verbatim — + `GGML_BLAS=OFF` forced, every GPU backend off, no `-DGGML_NATIVE` because + ggml's default is `-march=native` for the building host: the image is + built where it runs. Weights fetched by + `scripts/download-frank-tts-engine-qwentts.sh` with byte and LFS-oid + checks; the deployer gates on `--verify-only` rather than pulling 1.4 GiB + inside a deploy. + +**frank-tts: the 7772 endpoint gets its deploy wiring, and voice naming is written down** + + +- Stage 2 of the voice stack: the **façade** in front of the engine is now a + deployable service too. Runbook: [[Frank TTS]]. +- **Two attached networks, in a meaningful order.** `dankclaw_default` + first — it owns the default route and is where consumers live + (open-notebook dials `http://frank-tts:7772/v1` by container name) — then + the voice pair's private bridge. `scripts/frank-tts.env`'s `TTS_UPSTREAM` + moves from `host.containers.internal:8880` to + `http://frank-tts-engine:8880`, the actual fix for a shape that could + never have worked: a bridge container cannot reach a host-loopback bind, + and the engine publishes nothing. +- **`Wants=` + `After=` on the engine, never `Requires=`:** the façade still + carries the `llama-tts-cli` rollback backend, which needs no upstream, so + an engine that will not boot must delay this endpoint, not take it down. +- The façade handler installs the **engine's** quadlet pair as well — not + tidiness: with a referenced `Network=` file absent the v1 Quadlet + generator produces no container unit at all — and it `mkdir -p + /home/dank/voices` as the deploy user, because rootless podman + auto-creates a missing bind source **root-owned** and the endpoint then + reports "no voices" instead of an error. +- **A voice is a lowercase directory name.** `^[a-z0-9][a-z0-9._-]*$`, + ≤ 64 chars, no slash/`..`/space. The shape is the intersection of two + rules that do not agree: `app.py` would accept mixed case, `tts-server` + lowercases names — so a pack named `Narrator1` resolves through the façade + and not through the engine, and the boot warmer refuses such a directory + loudly. `narrator1/2/3` are load-bearing for open-notebook. +- **No Prometheus job, on purpose.** The façade exports no `/metrics`, + Prometheus 3 refuses HTML/JSON content types (the `comfyui` and `services` + probes are documented down for exactly that), and `LlmEndpointDown` is + scoped `up{job=~"frank-.*"}` — so a `/health` scrape would page on Signal + every few minutes for a healthy endpoint. That is the frank-archon + never-green lesson pointed the other way, written in the page instead of + quietly ignored. + +**voice-pack builder: the 10-second source-material gate becomes code** + + +- Third slice of the voice stack: `scripts/build-voice-pack.sh` turns a + source recording plus a crop offset into a deployable pack — + `reference.wav` (24 kHz mono `pcm_s16le`, 18–25 s), the engine's own + `reference.spk` + `reference.rvq` made from it, and `manifest.json` + provenance. Tracked tunables in `scripts/voice-packs.env`; new runbook + [[Voice Packs]]. +- **The gate is the product.** The project-wide invariant — no voice + material from the first 10 s of a sample file — had only ever been a + sentence in `scripts/frank-tts.env`. An offset inside it now exits 1 + before a directory is created, and the same assertion runs again against + the manifest that was just written, next to the recorded sha256, so a pack + cannot carry a forbidden crop even if the env file is edited into + nonsense. `10` stays a constant in the builder; the env file carries + offsets, never the law. +- **The extractor's contract was read, not guessed:** `qwen-codec` in the + pinned qwentts.cpp commit has **no `-o`** and names both artifacts next to + the input with the extension swapped. So the WAV is staged under its final + stem and `reference.wav` — the file the façade gates on — is moved into + place **last**, after the artifacts and the manifest: a failed extractor + leaves a directory that is not a voice. `--force` retires an existing pack + whole to `.AGENTTRASH/` instead of touching bytes; nothing in this script + deletes. +- Sizes are arithmetic, so they are checked: 16 codebooks × 11 bits = 22 B + per 12.5 Hz frame and `enc_dim 2048 × f32` = 8,192 B. That makes the + **effective crop window 19.84–25 s**, narrower than the WAV law's 18–25 s, + and a too-short crop is refused before the extractor runs. An engine bump + that changes either number fails the build by design. + +**voice stack audit: three faults that no gate could see** + +- The voice stack had never been run end to end, so the whole path was + audited as static code against the tools it calls, and three faults came + out of it. All three are the same species: a step that **reports success + while doing nothing**. +- **The build would not have happened.** `Containerfile.engine` created its + unprivileged user with `adduser`; the pinned ubuntu:24.04 base has no + `adduser` at all — proven in the base on Dankclaw, not argued from memory. + Replacement: `useradd --system --uid 10000 --home-dir /app --shell + /usr/sbin/nologin tts`; its `SYS_UID_MAX` warning is cosmetic and is the + price of matching the façade's uid. +- **The warm pass would not have warmed.** The unit hands its env file to + podman via `--env-file`, which performs no shell parsing and takes the + literal remainder of the line, while `source` strips one quoting layer. + `WARM_TEXT` therefore reached the container with its quotes, the JSON was + invalid, and every voice would log "registered, but the warm synthesis + failed" beside an `active` unit. Fixed in the hook; the bug survived the + first draft precisely because every existing test used `source`. +- **A rebuilt image would not have been rebuilt.** `build_compose_image` + skips `podman build` on a matching `cfhash` label, and that hash covered + the `Containerfile` alone — so editing `app.py` or `tts-engine-warm.sh` + (both `COPY`'d) skipped the build while the deploy reported success and + `/health` answered with old code. Each case arm now hashes the files its + `Containerfile` copies. +- Docs carry the lessons where an operator will hit them: a **Locally built + images** section in [[deployment]], two new troubleshooting lines in + [[Frank TTS Engine]] (all-voices-cold = quoted `WARM_TEXT`; façade 502 + beside 200 = model loaded ≠ voices registered), and `Voice Packs.md` + step 4 corrected — the warm hook re-registers **every** pack on every run, + because the engine replaces a name it already holds, so a restart is what + an engine bump needs and not what a rebuilt pack needs. + +**frank-tts-engine deployed: it builds, it serves, and the warm hook died on the kernel's argument cap** + +- Merged as **#14** (`747c665`) and deployed `--service frank-tts-engine`; + the image built clean — the first real proof of the `useradd` fix — and the + unit was `healthy` after 31 s. +- Then the deploy reported `0 voice(s) warm`: + `/app/warm-voices.sh: line 152: /usr/bin/curl: Argument list too long`. The + hook handed the registration body to curl as ONE argument; the kernel caps + a single argument at 131,072 bytes (`MAX_ARG_STRLEN`) and a shipped ~955 KB + `reference.wav` base64's past that, so curl never executed and the hook + logged that as "HTTP no response". Fourth silent fault of the voice stack, + same species as the other three: `/health` stayed green throughout. Fix: + the body goes to a `mktemp` file and curl gets `@file`; the regression test + is the honest shape — real listener, real `curl`, a 300 KB fixture pack — + so it fails with the same kernel error the host produced. Every shipped + pack sits well over the ~96 KB cliff, so no small fixture could ever have + seen this. +- **A trial law did not survive deployment.** Against an empty registry the + first synthesis of a freshly registered voice cost ~1.9 s, the next ~1.2 s. + The trial's 6.2–8.0 s cold / 0.5 ms restore ladder, which the warmer's own + header repeated, is retired: warming buys registration — mandatory, because + the registry is memory-only — plus about **0.6 s**. Corrected in + [[Frank TTS Engine]] law 1 and its troubleshooting table, and in + [[Local TTS and Voice Cloning]] and [[Voice Packs]]. +- Sweep afterwards: the retired ladder still lived in six places the first + pass missed — both quadlet files, `scripts/frank-tts-engine.env`, + `scripts/frank-tts.env`, the [[Frank TTS]] troubleshooting row, the + research brief's consequences paragraph, and one test comment. All + re-keyed to the measured claim. The trial entry earlier in this log is + left as written — a dated worklog entry is a record — and this line is the + correction that points at it. + +**7772 is live and the harness speaks through it** + +- Merged **#15** (`0315a62`, the warmer's argument-cap fix) and **#16** + (`d2ce669`, the comment sweep), then deployed both voice services in order + (`--service frank-tts-engine`, then `--service frank-tts`); Playwright gate + 33/33 twice. Deployed-state proof from a LAN client: `/health` → + `omni-http` with all three voices, and the same sentence in `narrator1` vs + `narrator2` came back with different sha256 and different RMS — the name + actually selects a voice. +- **The harness `speech` role moved off Kokoro**: `config.yml` + `modelRoles.speech: frank-tts (dankclaw)/frank-tts`. No voice key moves + with it. +- Two harness laws learned by breaking it first (both now on [[Frank TTS]]): + an explicit speech role has **no fallback candidate**, so a dead endpoint + errors instead of degrading; and `models.yml` loads at process start while + `config.yml` is read live, so a session older than the `frank-tts` row + fails exactly like a dead endpoint. +- Corrected the same evening, by breaking it in a restarted session: a + harness call that names no voice reaches 7772 with **no `voice` key at + all** — `400 unknown voice None; available voices: …` — because no setting + feeds one. The endpoint therefore grew `DEFAULT_VOICE` (its own PR): a + voice-less request speaks the declared pack, an unknown name still 400s + with the list. `speech.voice: narrator1` — briefly recorded as wiring — + went back to `am_fenrir`: that key is the vocalizer's Kokoro enum, so a + pack name there breaks speak-aloud and does nothing for synthesis. +- Status sweep: six pages still said the endpoint did not exist — both vault + descriptions, the two `index.md` bullets, the Open Notebook blocker (the + voice leg works; open-notebook itself is what is missing), the engine + page's deploy order (now a deploy record), and the homepage card, which + lost `wip: down` so the admin panel probes 7772 for real. + +**the default voice shipped: a harness voiceover needs no voice** + +- Merged **#17** (`b590695`) and **#18** (`7889a68`) and deployed + `--service frank-tts --service homepage` in one dispatch, Playwright gate + 33/33. The call that had failed is the one that proves the fix: same + device, **no** `voice_id` — `Saved 430124 bytes (…, voice=default, + codec=wav)` — and the façade journal says what actually spoke, + `speech voice=narrator1 (default)`. The harness prints `voice=default` + because it never had a name to print; the voice that answered is the + operator's `DEFAULT_VOICE` line in `scripts/frank-tts.env`, and the journal + keeps them distinguishable forever after. +- The page gained a **Default voice** row in its contract table and + [[#Naming a voice]] now states that a call naming nothing is the harness's + normal shape, not a client bug — the misconception that produced the 400 in + the first place. + +**OpenWebUI "voice narrator1 not found": the rejector is in the browser** + +- Frank's toast — `Error: Voice "narrator1" not found. Should be one of: + af_heart, …, bm_fable` — looked like frank-tts refusing the name. It is + not: that 29-name enum is the voice pack of **Kokoro.js running in the + page** — OpenWebUI 0.11.4 ships that implementation in its own front end. +- Instruments that separated the two paths, in order: the admin config DB is + correct (`audio.tts.engine=openai`, base `http://host.containers.internal:7772/v1`, + `voice=narrator1`); from inside the container the exact OpenWebUI body + returns **200 `audio/mpeg`**; and `podman logs open-webui` holds **zero** + `audio/speech` requests for the failing click. A server-path failure leaves + a log line and a frank-tts error — a `browser-kokoro` failure leaves + neither — and no container env or `user.settings` row carries an override, + so the engine choice is per-browser storage, not fleet state. +- Fix is a UI click, not a deploy: Settings → Audio → TTS Engine → + **Default** hands synthesis back to frank-tts and the cloned voices; + choosing `af_heart` keeps in-browser Kokoro, which cannot clone (fixed + pack). Recorded on [[OpenWebUI]] under *Text-to-speech (frank-tts)*. +- Also in the window: **PR #22 merged** (`0c7fe8b`, `dank/command-center`) so + `agent/extensions/dankvoice.ts` landed on the auto-load path — proven in a + plain `omp` session with no flags: `/say merged build check` → WAV to the + player, temp dir cleaned. Sessions open before the merge need a restart to + pick the command up. + +**frank-zealot: 2 × 262,144 over unquantized f16 KV** + +- 2026-10-18 — frank-zealot: shape cut to 2 × 262,144 over unquantized f16 KV + (the 786,432 pool cannot fit exact KV on the 3090; concurrency 3 → 2, and + MTP is now doubly out — off by the 13:16 measurement, and exact KV bills a + draft-context f16 pool over the whole context). See [[frank-zealot]]. +- Arithmetic correction to the repo's long-standing figure: "the f16 pair + would need **26,112 MiB**" doubled the q8_0 number *including* its 1/32 + scale overhead — q8_0 is 17,408 B/token (1.0625 B/element), so f16/q8_0 is + **1.882×**, not 2×, and exact f16 over 786,432 is **24,576 MiB**. The + conclusion stands (three f16 windows cannot boot this card); the number was + wrong. +- New gate in `tests/frank_zealot_unit.bats`: dtype × pool plus the measured + non-KV terms must fit the driver-addressable card with **≥ 1 GiB** of slack + — and the same gate proves it refuses the impossible arm (f16 at 786,432), + not merely passes the shipped one. + +## 2026-10-17 +**frank-zealot: Bonsai-2-27B PQ2_0 trial + cutover (2 × 200,704, q8_0 KV)** + + +- 2026-10-17 — frank-zealot: **Ternary-Bonsai-2-27B (PQ2_0) cutover**, two + slots at 200,704 over q8_0 K + q8_0 V on the RTX 3090, fetched by + `scripts/download-frank-zealot-bonsai2.sh` (exact bytes + sha256, + `--verify-only`). See [[frank-zealot]]. +- Runtime door: the file is `file_type 141` with 402 tensors of custom GGML + type **142** plus `prism.hadamard.*` metadata — stock llama.cpp refuses + it, so the PrismML vendor build is **forced, not chosen**; the vendor + build serves q8_0 V under flash attention, so the source rebuild with + `GGML_CUDA_FA_ALL_QUANTS=ON` was cancelled as unnecessary. +- Trial-window law cost one boot: the scratch env invented + `KV_CACHE_TYPE` where the serve script reads + `CACHE_TYPE_K`/`CACHE_TYPE_V`, so the script's `:=f16` default silently + won and every arm booted f16. A silently-defaulted knob turns a whole + trial window into a different experiment. +- New fleet laws recorded (AGENTS.md + runbook): the mmproj's worst-case + VRAM is NOT inside the engine's fit projection; the first request after a + boot is a warmup artifact; a 64-token thinking budget makes a thinking + model look blind; the Bonsai-2 template reads `reasoning_effort` + (xhigh/medium/low, raises otherwise) so the harness effort fold is + load-bearing on this endpoint, and it reads `preserve_thinking`. + Recurrent DeltaNet state bills ~150 MiB **per slot** — pool arithmetic + alone misses it. +- Base alignment: **Frank chose the OFFICIAL censored payload** — publisher + provenance over refusal-freedom. The 9B it replaced was uncensored, so + the homepage card and its warning contract were re-keyed: zealot's card + now carries the base-alignment notice, the shared legal warning belongs + to dragoon alone. The CRACK re-export (byte-identical geometry per header + census) stays on disk as the one-line `MODEL_FILE` fallback if refusals + bite. +- Two live-path stragglers the cutover had to catch: + `serve-frank-zealot.sh` still DEFAULTED to the retired payload's paths + (a defaulted run would have served the old model — now re-keyed to the + shipped trio), and the OpenWebUI `model` row's `meta.description` is live + DB state (re-keyed with a DB backup; no `open-webui` restart forced — + forcing it logs every family session out for one cosmetic string). + +**dankserver disk purge: Swift w4dn8 rollback + orphan caches (docs re-key)** + + +- Ask: "dump the swift rollback and the GLM GGUF — make sure the model + caches and agenttrash are empty as well." Preflight first: `lsof` empty + on both payloads, the serving trio names only the live path, and a podman + layer census proved no deleted path backs an image container (the 121 GB + layer is the ENGINE image, not a model blob). Staged, then purged with + Frank's approval: the two retired payloads, derived kvcache tiers, stale + compile caches, and every `.AGENTTRASH` on the host. +- Recovery: the retired w4dn8 rollback payload is byte-exact on the Hub — + rollback is now a **re-download, not a local file**. Corpora (grams, + kld-ref, calibration JSONL) deliberately KEPT: they are the re-convert + and KLD-gate inputs. +- Post-checks: frank-archon active and `/health` ok, serving payload + byte-size intact, the live kvcache tier untouched, no process references + a deleted path. +- Docs re-keyed: AGENTS.md fleet row + frank-archon section + both runbook + index rows; the live runbook's Rollback rewritten (HF re-fetch procedure — + the fetch script's REPO/RAD_NAME/STAGE/DEST/SIZE/SHA256 constants are + hardcoded, so retargeting at the mirror is a script edit); the retired + runbook's header corrected (was still "LIVE", never updated at + retirement). + +**frank-dragoon cutover: native v2 abliterated payload, card's memory-saving pool** + + +- 2026-10-17 — frank-dragoon: cutover to the native v2 abliterated halogen + payload with the card's memory-saving KV pool; census first: the n-gram + table was ALREADY disk-backed and the only real lever was the KV pool. + See [[frank-dragoon]]. +- **Filename trap caught before the fetch:** this repo name carried two + different artifacts 32 bytes apart (the writer-transplant era's file vs + the native conversion). Byte-exact gate + LFS sha256 in + `scripts/download-frank-dragoon-v2abl.sh`, which also HARDLINKS the + peonist n-gram/vision/tokenizer trio — the release publishes only the + main file, and a hardlink is a regular file, so the no-symlink serve gate + passes without a 47.7 GiB copy. +- **Companion risk adjudicated by the engine, not by argument.** The card + does NOT establish the stock peonist companions as matched to a native + conversion and publishes no source-matched pair, so the A/B ran the + checkpoint tools on a stopped service under one knob set + (`scripts/census-frank-dragoon.sh`) plus a behaviour battery + (`scripts/battery-frank-dragoon.py`): two sound models disagreeing on + content, not a corrupt forward pass. The boot window ran through a + transient `FRANK_DRAGOON_ENV_FILE` drop-in — never a scratch port (one + iGPU, one server) — with the stop+mask law and a trap-guarded restore + (`scripts/trial-frank-dragoon.sh`). Masking fails on a relinked unit + file, as documented; stop alone was enough. +- Cost of the swap, stated plainly: two MAXIMUM-length conversations no + longer share the endpoint, the finetune's perplexity edge is gone, and + its reasoning came out compressed ~1.9×. Bought: RAM back as page cache + for the disk-backed n-gram table, faster cold prefill, clean retrieval, + and the base checkpoint itself instead of the finetune. The seam's pool + law relaxed from `pool == slots × ctx` to `ctx ≤ pool ≤ slots × ctx`. + +**frank-dragoon: ceiling and pool cut to 200,704; crawl reset fix** + + +- 2026-10-17 — frank-dragoon: context ceiling and KV pool cut to 200,704 + (49 × 4096, the per-slot rung frank-zealot runs; nothing in the fleet + needed the bigger window). See [[frank-dragoon]]. +- Harness moved WITH the ceiling (400-zone law): `models.yml` + `contextWindow` = ceiling − `maxTokens`; compaction fires at 85 % of the + window, and the 180k compaction target of the 262,144 era is unreachable + because the window itself is smaller. Seam re-keyed to prove the + invariant at the new pair — `ctx ≤ pool ≤ slots × ctx` and + `harness window ≤ ctx − maxTokens` — so a future raise of one without the + other fails the gate. +- Probes on the live endpoint: over the ceiling → `400 … tokens over` in + 0.86 s — preflight rejects before any prefill, so probing costs the + endpoint nothing. +- Deploy verdict: the endpoint dispatched healthy (`/health` 200), but the + `[3/3]` Playwright gate failed on `12-crawl` and the deployer correctly + said **DEPLOY INCOMPLETE**. Root cause was NOT the endpoint — it was Plex + Web, signed out in the `qa-e2e` profile, running its own sign-in flow, + which broke the spec two different ways: (1) it redirects the top frame, + so Chromium cancelled the crawl's in-loop `page.goto("/")` with + `net::ERR_ABORTED` — and that reset sat outside the `try`, so one host + aborted the whole crawl; fixed with `backHome()`, which re-issues on that + one error class. (2) The same hand-off leaves the browser on + `app.plex.tv` (Plex's hosted surface for a session-less client), so the + `hostname == target` law could never settle; fixed with an explicit + `HANDOFFS` landing list. Plex itself was healthy throughout, and the + guard contract is still asserted for every vhost. Both symptoms recorded + as gotcha 6; the gate after both fixes went 33 passed with the crawl + green on the first attempt. + +## 2026-10-15 +**frank-archon: publisher iq4r payload swap (MODEL_FILE-only)** + + +- 2026-10-15 — frank-archon: publisher iq4r payload swap, `MODEL_FILE`-only + — `StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe`, the engine author's + published rad-convert of the OFFICIAL Qwen3.8-Flash-Next, **no other + argument changes**, mindful the operator session runs on the endpoint + being swapped. See [[frank-archon]]. +- Disk: Frank authorized one `rm` — the spent BF16 source and the + two-era-old fp8-native `.rad` left `/home/dank/models`; fp8-native is now + rebuild-only (fetch + build scripts); rollback for THIS swap is the + previous `.rad`, on disk, by reverting the commit. +- Fetch: new `scripts/download-frank-archon-iq4r.sh` (hfget, exact bytes + + Hub LFS sha256, `--verify-only` default) as a transient unit. The cutover + **self-severed by design** — a detached `deploy.sh` survived the session + death when frank-archon restarted; engine healthy in ~100 s, and the + prefix tier RECOVERED across the swap (arch+tokenizer unchanged). +- e2e gate went red on three dankclaw specs — NOT the swap: dankclaw + rebooted 01:41 and the gluetun boot-race dependency-failed `searxng`, + `marreta`, and `marreta-browser` (jobs consumed while gluetun's start job + retried; `Restart=always` never armed because the services never ran). + Cascade restarted; skill codified + (`gluetun-dependent-boot-race-recovery`). frank-archon's own health gate + passed every deploy. + +**authentik: two more human accounts (Dankclaw)** + + +- **Live — mitch (pk 22) and eric (pk 23) provisioned via `ak shell` ORM, + no repo objects**, each `@franksdank.cloud` in the human-user + template shape (password via `set_password` on stdin only — never argv, + so nothing lands in `ps`), verified through the exact PasswordStage code + path; zero `proxysession` rows for either, so no claim purge was due. +- `/serverAppConfigs-docs/Infrastructure/Authentik.md` — human-user census + extended with pk 22–23. + ## 2026-10-13 - **frank-archon outage: a kernel knob, not the model.** The 2026-10-12 23:17 power failure on dankserver reset `/sys/module/ttm/parameters/pages_limit` @@ -117,6 +887,28 @@ tokens, per-stream + aggregate) so no future shape change can be priced on prefill alone. +## 2026-10-11 +**frank-archon: unquantized (bf16) KV at 2 × 200,000** + + +- 2026-10-11 — frank-archon: **unquantized (bf16) KV at 2 × 200,000** on + exact attention state, with pool, slab and host pool rebalanced around + the bigger cache. See [[frank-archon]]. +- Engine door: `--kv-cache-dtype` takes ONLY `bf16|fp8` — one dtype for the + whole cache, so llama.cpp's exact-K / quantized-V pair (what frank-zealot + runs) has **no radiance equivalent**. +- `--max-num-seqs 3 → 2` booted clean: the fleet's "seqs 2 cannot boot" law + is the retired tcclaviger vLLM fork's torch.compile `query_start_loc` + constraint, not radiance's — radiance booted seqs 2 in its fp8-native era + and the sweep had 2 and 3 tied inside noise. +- Harness law re-learned: radiance puts the thinking block in + `reasoning_content`, so a probe that reads `content` on a small budget + sees empty text with `finish_reason: stop` — a healthy server scored 0/2 + until the probe set `chat_template_kwargs.enable_thinking=false`. Tool + committed as `scripts/probe-frank-archon-kv.py`. +- The fp8-vs-bf16 control arms and the open keep-bf16-or-take-cheap-fp8 + call are recorded on [[frank-archon]]; the 2026-10-13 section below (in + file order above) closes that call in favour of bf16. ## 2026-10-10 - **Fleet service rename: the StarCraft names became the SERVICE names.** @@ -504,6 +1296,243 @@ positive, KV pool >= 400,700 tokens at the measured 7,892 B/token). Runbook: `serverAppConfigs-docs/LLM/frank-archon.md`. +## 2026-10-02 +**frank-dragoon cutover: vLLM MiMo -> ROCmFPX Q8_0_ROCMFPX_AGENT + MTP** + + +- 2026-10-02 — frank-dragoon: vLLM MiMo → ROCmFPX `Q8_0_ROCMFPX_AGENT` + payload + MTP from the repo's own bundled head, one session at the native + 262,144. See [[frank-dragoon]]. +- The runtime is **forced by the file**: tensor type 103 + (`GGML_TYPE_Q8_0_ROCMFPX`, ftype 115) is unreadable by stock llama.cpp, + gufo and every other loader on the fleet — only the kingjones30/ROCmFPX + fork can read it, so the engine is pinned to commit + `36e9acd40e10a87cd3c3ef8ec734668757dc8520` and built by + `scripts/build-rocmfpx.sh`. The AGENT routing variant was chosen over the + card's plain Q8_0 because it keeps 187 output-head tensors at plain Q8_0 + instead of 2, which measured better MTP acceptance. MTP was the whole + performance story of the shape (roughly +120 % decode, ladder on the + runbook). +- Rollback: git tag `frank-dragoon-gufo-q4kxl-20261001`; the retired + vLLM-era fetchers moved to `old/`. Runbook [[frank-dragoon]]. + +**frank-zealot: Q5_K_M, then three contexts + vision, sized by two boots** + + +- 2026-10-02 — frank-zealot: Q5_K_M payload, then three contexts + vision + in one day, sized by two boots (`CTX_SIZE=540672 PARALLEL=3`, f16 K + + q8_0 V under flash attention, `--mmproj`). See [[frank-zealot]]. +- **The pool came from two boots, not a formula.** Differencing the boots + prices the non-KV floor — and prices the vision tower at **~1.4 GiB for + a 918 MB file**: the encoder's own buffers ride with it. The first 3-slot + deploy booted healthy but left less than the ordered 2–3 GiB band free, + which is why the seam now prices non-KV at the measured value and + REJECTS that shape. +- Three contexts cost window, and that is the tradeoff Frank was told: + 180,224 tokens per conversation, not the native 262,144; a longer prompt + is a hard `400 exceed_context_size_error`, so `models.yml` follows the + slot, not the card. +- **Vision contract gotcha:** `/v1/chat/completions` ignores the legacy + top-level `images` field — that request answered "Brown." for the same + red image, i.e. it generated from text alone. Only the `image_url` + content part reaches the model. `/completion` accepts `images` but this + build still requires `messages`. +- Deployer gates on the projector too (`LLM_SMALL_MMPROJ`), so a missing + mmproj cannot silently ship a blind endpoint; `MMPROJ_FILE=` + (explicitly empty) in the env file is the documented text-only opt-out + and returns the tower's ~1.4 GiB to KV. +- Reclaim staged, **not purged, awaiting approval**: + `~/.AGENTTRASH/20261002-retired-llm-payloads/` of superseded payloads. + Runbook [[frank-zealot]]. + +**frank-archon shape D: 3 x 200704, fp8, pool sized for 2 contexts** + + +Frank's order: shrink frank-archon to a 200704-token window with about two +resident full-length contexts, cut the KV pool, and drop MNBT to 1024. The +first attempt (`30904ba`, 2 x 200704 + SEQS=2) **crash-looped**; the revert +went live as `e460bbb`, and the shape was then re-applied one law at a time on +top of the proven-good base, root cause first. + +- **ROOT CAUSE: `--max-num-seqs 2` cannot compile this model. It was never + OOM, never MNBT, never the KV dtype.** Four single-variable boot probes + against the then-deployed shape (production stopped for each, candidate + booted AS the service): 262144 x SEQS 2 fails at BOTH MNBT rungs with + `torch.fx.experimental.symbolic_shapes.ConstraintViolationError: + Constraints violated (L['query_start_loc'].size()[0]) ... specialized it + to be a constant (3)`, raised from AOT `build_guards(strict_error=True)` + during `determine_available_memory -> profile_run -> _dummy_run`, ~7 min + in; SEQS 3 is healthy at both rungs. `model_runner.py` sizes that input + buffer `max_num_reqs + 1` and `_dummy_run` sets `num_reqs = + min(max_num_batched_tokens, max_num_reqs)`, so its width is **SEQS + 1** + for ANY `MNBT >= SEQS` — 3 at SEQS 2, exactly the constant in the + message; MNBT only moves the compile RANGE, which is why the identical + error appears at both rungs. `--enforce-eager` bypasses the path but + drops CUDA graphs. The 2026-10-01 bf16 2 x 262144 failure was this same + error class; its message named "KV dtype and SEQS" together and nobody + bisected, so the dtype was blamed — it was a red herring. +- **THE ARENA LEVER IS `max_model_len`, NOT `--kv-cache-memory-bytes`.** + The planner sizes expert cache as `budget - non-expert - KV reserve - + graphs - overhead`, and the KV reserve is registry-priced at **9190 + B/token x max_model_len**, independent of the pool override: d(reserve) + equalled d(arena) exactly, and the multi-GB pool cut bought **zero** + arena slots. The previous session's premise — "the pool cut becomes + arena, ~7,500 slots" — was wrong; the pool cut is a runtime-HEADROOM + lever (the shape it replaced died with `0 bytes free` on a real + 995-token request). +- **Final shape (option D):** the argv keeps the only slot count that + compiles; the POOL caps resident conversations (ledger on + [[frank-archon]]). Measured live, one prediction corrected: the arena + prediction landed exactly, the concurrency one did not — **the per-token + KV price is shape-dependent** (7,701 B/token at MAXLEN 262144 vs **7,892** + at 200704, ~2.5 % dearer). Never extrapolate a concurrency figure across + geometries. Smoke probes confirmed the 3 slots real, native `tool_calls`, + and that `reasoning_effort` accepts only {xhigh, medium, low}, answering + 400 for anything else. +- **Harness contract followed the window:** `models.yml` `contextWindow` + 262144 -> **200704**; side effect: frank-archon's compaction request fits + inside a frank-zealot slot again. A comment-only follow-up `295d578` was + deployed so the per-service revision marker sits at HEAD. Rollback for + the pre-shape-D fp8+MTP4 era stays git tag + `frank-archon-fp8-mtp-20261001`. Runbook [[frank-archon]]. + +**plex: the readiness gate probed HTTP against a TLS-only service** + + +Found by accident, and it had been latent since the secure-connections posture +was set: `deploy_plex` gated on `curl -fsS http://127.0.0.1:32400/identity`, +which **cannot** succeed. PMS drops plain HTTP off-loopback - the probe's +connection arrives on the container's eth0 - so Plex accepts the TCP connection +and then answers with nothing, and curl reports "Empty reply from server" (52). + +- Trigger: a bare `./scripts/deploy.sh dankserver` (no `--service`) drains + the whole pending backlog, not just the service you were thinking about. + plex's per-service revision had been behind long enough that this gate + had never run against the current posture; it burned its full 600 s and + failed the run, and every service after plex in the order was left + undeployed. +- **Plex was healthy the entire time.** `https://127.0.0.1:32400/identity` + returned 200 with the MediaContainer XML in 44 ms while the gate was + still timing out; the in-container plain-HTTP path also answers 200 — + loopback is the one path PMS serves in the clear, which is exactly why + the host-side probe could not see the difference. +- **The repo already knew this law, in a different file:** + `tests/caddyfile.bats` pins the plex vhosts to a TLS upstream with the + comment "PMS drops plain HTTP off-loopback". The deployer's gate never + got the same treatment. Fixed to `curl -fsSk --max-time 3 + https://127.0.0.1:32400/identity` (`-k`: PMS serves its own self-signed + cert on 32400). Re-deployed `39f0e0d`: plex recorded deployed, e2e + 33/33, frank-archon untouched (`health=200`, NRestarts=0). +- **Operating lesson: `deploy.sh ` with no `--service` is a + fleet-wide drain.** It restarts every service whose recorded revision is + behind HEAD and whose paths changed since. To touch one service, name it. + +**frank-archon shape E: `--expert-precise-memory` (+1,026 arena slots), a mounted compile cache, and the P2P answer** + + +Shape D left one question open — the plan's overhead term was reserving 2.35 GiB +of compile headroom the boot never used — and the answer to Frank's free-VRAM +question arrived together with a boot-cost bug that had been costing 181 s on +every restart since the cutover. + +- **`--expert-precise-memory` is real and it fits (option A).** Boot-probed on + the shape-D base with production stopped, the candidate booted AS the service + (`FRANK_ARCHON_ENV_FILE=/tmp/probe-b.env bash scripts/serve-frank-archon.sh`, + trap-guaranteed restore), the numbers read off the engine's own plan lines: + + | plan line | shape D | shape E | + |---|---|---| + | `runtime overhead (non-torch + buffers + activation)` | 7.60 GiB | **5.25 GiB** (`precise (gfx1201 TP2 nbt4096/seqs16/len262144)`) | + | `expert cache space` | 14.87 GiB | **17.22 GiB** | + | slots | 6,496 | **7,522** (+1,026, +16 %) | + | `swap region` | — | 13.7 GiB / 5,986 slots, 133-134 per layer | + | boot | — | **healthy in 420 s**, no OOM | + + The flag also prints `'expert_precise_memory': True` in `non-default args`, so + a silent argparse drop cannot hide. It was refused on 2026-10-01 only because + the 2 x 262144 bf16 shape was already 0.13 GiB under the ceiling — there the + flag was the last stack over the line, not the cause. +- **The pool was NOT cut (option A) because the pool is not an arena lever.** + A 2.2e9 arm was probed in the same window and reports `277,686 tokens, + 1.38x` — measured, rejected as the shipped value, kept as the conservative + fallback. With 3.15e9/rank the endpoint keeps shape D's 1.99x concurrency and + gains the arena, which is the shape D follow-up had actually been looking for. +- **THE COMPILE CACHE WAS BEING THROWN AWAY ON EVERY RESTART — 181 s each.** + Measured: a fresh container has **no `/root/.cache` at all** (`No such file or + directory`); after our boot it holds `/root/.cache/vllm` at + **1,154,714,333 B in 4,665 files** plus `/root/.cache/comgr` at 229 MB, inside + the ephemeral layer, and `podman rm -f` (every deploy, restart, reboot) + discards them. The engine names the loss itself: `Compiling a graph for compile + range (1, 1024) takes 173.19 s` ... `torch.compile took 181.20 s in total` ... + `init engine (profile, create kv cache, warmup model) took 217.26 s + (compilation: 181.20 s)`. The serve script now mounts + `FRANK_ARCHON_CACHE_DIR` at `/root/.cache` — the **parent**, so the vLLM + AOT/torch_compile tree (`VLLM_CACHE_ROOT`, default `/root/.cache/vllm`) and + the comgr JIT cache both persist, and nothing is hidden because the image has + no such directory. Same class of fix as the recipe's + `/cache/{triton,vllm,inductor}` volumes; the compile RANGE is part of the + cache key, so an MNBT change costs one recompilation, not a permanent tax. +- **`GPU_MAX_HW_QUEUES=2`** (HSA hardware queues per GPU) added at Frank's + direction. Verified a real knob rather than a no-op — the literal is read from + `/opt/rocm/core-10.0/lib/libhsa-runtime64.so` — and shipped deliberately + alongside the 32x64 gather geometry. **Not A/B measured.** +- **P2P: the hardware supports it, and the engine is already using it.** The + live log shows `r4d all-reduce INSTALLED (rank=0 max=49152KB finegrained=T…)` + on BOTH ranks and `Using ['R4D_AR', 'PYNCCL'] all-reduce backends` — R4D's + P2P path, with RCCL as fallback, measured at **45.274 GB/s bidirectional**. + Root ports `00:01.1`/`00:01.3` both report ACS `ReqRedir+`, so P2P hairpins at + the root complex; there is no XGMI. The old FAIL predates the 2026-09-20 board + swap. Kernel support confirmed: `CONFIG_HSA_AMD_P2P=y`, `CONFIG_PCI_P2PDMA=y`, + `CONFIG_HSA_AMD=y`, `CONFIG_ZONE_DEVICE=y`, `iommu=pt` on the cmdline. +- **Recipe diff (Frank's 3 x R9700 compose vs this podman run), ranked:** + - *Acted on:* persistent compile caches (above — the one real operational + gap), and `GPU_MAX_HW_QUEUES`. + - *Retracted:* pinned `--compilation-config cudagraph_capture_sizes` was on my + list; measured, it is **not** a gap here — capture is 3 PIECEWISE + 2 FULL + graphs, `Graph capturing finished in 3 secs, took 0.31 GiB` (0.11 GiB on the + lighter rank). No 15-52-size ladder to cut. + - *Not acted on — measured candidates, none shipped (no bench window; these + are CPU-side and our decode is gather-bound, so a spin-wait stealing cores + from the gather threads is plausible but unproven):* `OMP_NUM_THREADS=8` + (defaults to all 16 logical CPUs on this 7700X), `HSA_ENABLE_INTERRUPT=1`, + `HSA_ENABLE_MWAITX=1`, `HSA_ENABLE_SDMA`, `HSA_NO_SCRATCH_RECLAIM`, + `HSA_ENABLE_IPC_MODE_LEGACY`. All six HSA names verified real — the literals + are read out of `/opt/rocm/core-10.0/lib/libhsa-runtime64.so` — so they are + knobs, not no-ops; any of them needs its own A/B before shipping. + - *Deliberate divergence, keep ours:* no MTP (the recipe has it; measured + fastest here), `qwen3_xml` vs `qwen3_coder` (ours verified live), no + `--override-generation-config` (server-side sampling defaults), published + port + Caddy rather than `network_mode: host`. + - *Non-gaps:* `shm_size 32gb` (ours is `--ipc=host`), `memlock -1` (same), + `SYS_PTRACE` (same), `/dev/kfd` + `/dev/dri` (same), `--enable-chunked-prefill` + (V1 default), `plecache` volume (the flag is not even passed), and + `ROCR_VISIBLE_DEVICES=1,2,3` vs our `HIP_VISIBLE_DEVICES=0,1` — verified + equivalent here: `rocm-smi --showid` puts the two R9700s at GPU[0]/GPU[1] and + the 7700X iGPU at GPU[2], so our indices are the right two cards. +- **Harness note, because it cost two probe attempts:** a probe that reported an + instantaneous candidate death was a logging artifact of the inline SSH script, + not a crash — the container and its serve script must be polled by recorded id + and their logs dumped on the exit path, or a failure looks like a bad flag. + The flag was fine both times. +- Live: `health=200`, `NRestarts=0`, arena 17.22 GiB / 7,522 slots, KV 398,658 + tokens / 1.99x. Docs: [[frank-archon]]. +- **Deployed `1ac4d48`, e2e 33/33, and the live plan matched the probe exactly:** + `'expert_precise_memory': True`, `runtime overhead … 5.25 GiB`, `expert cache + space 17.22 GiB / 7522 slots`, `swap region 13.7 GiB / 5986 slots`, `GPU KV + cache size: 398,658 tokens … 1.99x`, pool 3,150,000,000, MNBT 1024. The + precise overhead and the arena growth are now production numbers, not a + probe's projection. +- **The compile-cache fix was then proven on the same endpoint, two consecutive + boots of the deployed shape:** cold `Compiling a graph for compile range + (1, 1024) takes 179.24 s` → `torch.compile took 188.26 s`; warm `Directly load + the compiled graph(s) for compile range (1, 1024) from the cache, took + 2.143 s` → `torch.compile took 3.97 s in total` → `init engine (profile, + create kv cache, warmup model) took 41.13 s (compilation: 3.98 s)`. The host + dir held 1.4 GB in 4,707 files after the cold boot. **Time to `/health` is + dominated by the 172.8 GiB payload load** (the warm boot still needed 490 s), + so the honest claim is ~184 s of inductor/AOT build recovered per restart — + not a proportional boot cut. + ## 2026-10-01 - **frank-archon rolled BACK to Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4; GLM/DFlash2 declared failed and purged.** The DFlash2 drafter booted and @@ -523,6 +1552,276 @@ KV-cache patches 02/03/04/04b survive in `scripts/patches/` (generic unifier/tailpin/grouping/registry fixes, apply-verified). +**frank-archon RESTORED to fp8 3 x 262144 (bf16 boot fails)** + + +bf16 (--kv-cache-dtype auto) at 2 x 262144 did not stay up, with OR without +--expert-precise-memory: + +- with precise: NRestarts 1->3, each attempt ~6.7 min then + `Deactivated successfully` (API server drains after an EngineCore fatal + error) — cause not captured, the serve script rm'd the container. +- without precise (default 7.60 GiB overhead, arena 14.3 GiB): NRestarts + climbing, container `Exited (1)`, and the journal finally showed the real + error: + `RuntimeError: Worker failed with error 'Constraints violated + (L['query_start_loc'].size()[0])! You marked L['query_start_loc'].size()[0] + as dynamic but your code specialized it to be a constant (3)'` + — a torch.compile/dynamo dynamic-shape constraint violation, NOT an OOM. + `~/.cache/vllm` holds only `modelinfos` (16K) and each `podman run` gets a + fresh container filesystem, so a stale compile cache is ruled out: the + conflict is produced by this configuration's own compile path. + +Differences from the last shape measured serving (fp8 3 x 262144, MNBT 2048, +KV 6.1e9 -> 792,069 tokens / 3.02x / arena 6,248 / 3.59 GiB free): KV dtype +(fp8 -> bf16) and SEQS (3 -> 2). Both are candidates for the dynamo trigger; +`--enforce-eager` is the bypass that would sidestep the constraint path. + +Restored: fp8 3 x 262144, MNBT 2048, KV_BYTES 6,100,000,000, no precise flag. +Next: bisect bf16 vs SEQS in a window (bf16 + SEQS=3 isolates the dtype). + +**frank-archon: --expert-precise-memory + bf16 = boot crash loop** + + +**Reverted.** The two changes Frank asked for were applied together and the +result crash-looped: `NRestarts` climbed 1 -> 2 -> 3, each attempt running +~6.7 minutes then logging `Deactivated successfully` (vLLM's API server drains +and exits 0 after an EngineCore fatal error, so `podman wait` returns 0; the +new `Restart=always` turned a silent death into a visible loop). + +The budget arithmetic explains it without needing the error text, because the +arena is DEVICE-resident (the fp8 3-window shape's plan predicts 28.27 GiB and +that is exactly what `rocm-smi` measured): + +| line | bf16 2 x 262144 + precise | fp8 3 x 262144 | +|---|---|---| +| non-expert weights | 6.28 | 6.28 | +| pinned pool | 6.80 | 5.68 | +| CUDA graphs | 0.55 | 0.55 | +| arena | 16.65 (precise: overhead 5.25) | 14.31 (default: overhead 7.60) | +| non-torch | ~1.45 | ~1.45 | +| **total** | **31.73 of 31.86 GiB** | 28.27 (measured) | +| free at boot | **~0.13 GiB** | 3.59 | + +`--expert-precise-memory` hands the 2.35 GiB compile/graph-capture headroom to +the arena, and bf16 simultaneously adds 1.08 GiB of pinned pool; together they +put graph capture ~0.13 GiB from the ceiling, where every transient allocation +dies. So the flag is dropped and bf16 2 x 262144 ships with the default +overhead: arena 14.31 GiB, ~2.4 GiB free at boot. + +**Lesson:** the registry's "compile headroom" is not slack to be reclaimed by +default — it is the budget for the phase that allocates last. Reclaim it only +by paying for it elsewhere (smaller pool, fewer sessions, or a shorter context). + +**frank-archon: 2 bf16 windows + --expert-precise-memory** + + +Frank's final shape for the night. Two changes at once, both measured in +principle before the boot: + +- **KV is bf16 (`--kv-cache-dtype auto`), two full-size windows (2 × 262144).** + bf16 measures **13,839 B/token** (from the bf16 3×241664 boot: + 11,047,010,304 B -> 798,280 tokens), decomposing as **12,276 attention + + 1,563 mamba/side caches** against fp8's **6,138 + 1,563 = 7,701**. So + 524,288 tokens needs 7.256e9 B; pinned at 7,300,000,000 (~527,500 tokens, + 3,206 margin). Cost: the pool goes 5.68 -> 6.76 GiB per rank. +- **`--expert-precise-memory`.** The registry's default overhead for this + architecture is 7.60 GiB = the fitted 5.25 GiB (gfx1201 TP2 + nbt4096/seqs16/len262144) + 2.35 GiB of compile/graph-capture headroom. + The flag selects the fitted figure, which moves only the plan's `room` term + — arena 6,248 -> ~7,278 slots (~1,025 more, at 2.344 MiB/slot) — at the + cost of boot-time capture slack. The overhead formula adds 0.22 GiB per + 1024 nbt above 4096, so MNBT 2048 pays nothing. + +Budget note worth keeping: the arena is `(budget - non_expert_weights - +kv_reserve - graphs - overhead) // expert_bytes`, where `kv_reserve` is the +planner's OWN one-max-model-len reserve (2.48 GiB at TP2) and does NOT scale +with SEQS. The pinned `--kv-cache-memory-bytes` therefore does not shrink the +planned arena; it comes out of free VRAM instead. + +**frank-dragoon: Q4_K_XL payload + 4 x 163840 sessions** + + +Frank's call. Payload swapped from ajgazin's **abliterated** Swift-1.5 +UD-Q8_K_XL (29.3 GiB) to **unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL** (16.35 GiB, +sha256 3f227079…b01e, vanilla/Apache-2.0 — the endpoint is no longer +uncensored). New fetcher `scripts/download-frank-dragoon-q4.sh`; the DFlash2 +drafter and its fetcher are unchanged. + +Shape: **`--sessions 4 --context 163840`** with unquantized KV. gufo has no +KV-quantization lever at all (`execution_policy.hpp` pins `kv_cache_storage` +to `kFp16`), so "unquantized" is inherent. + +KV arithmetic (why this fits where one 262144 session used to sit): +f16 KV costs **64 KiB/token** — 16 full-attention layers (the hybrid is +`full_attention_interval: 4` over 64 layers) x 4 KV heads x 256 head_dim x +2 (K,V) x 2 bytes; the other 48 layers are linear-attention with a +per-SEQUENCE fp32 state (~144 MiB), not a per-token cache. 4 x 163840 = +655,360 tokens = 40.0 GiB of KV, and each session is exactly 10 GiB. Budget: +weights 16.35 + KV 40.0 + drafter 1.07 + linear state 0.56 + runtime ~4 = +~62 GiB against the 114 GiB GTT. + +**e2e gate flakiness (12-crawl) — do not panic at the banner** + + +`specs/12-crawl.spec.ts:23` ("every user open link reaches its app") failed the +deploy gate twice and then passed cleanly on a manual re-run: +- run 1 (deploy): `Expected "plex.franksdank.cloud"` / `Received "app.plex.tv"` + — Plex redirects its web UI to the hosted app; the same failure appeared on + the previous deploy too. +- run 2 (manual): `page.goto: net::ERR_ABORTED at https://www.franksdank.cloud/` + — the homepage answered `http=200` in 40 ms from the dev box at the same + moment, so this is a browser-level flake. +- run 3 (manual): **33 passed**, exit 0. + +So a red gate on this spec alone is NOT proof the fleet is broken, and the +deployer's `DEPLOY INCOMPLETE` banner can be raised by a flake. Re-run +`tests/e2e/run.sh` before treating it as real; the Plex redirect is the one +stable underlying issue and it lives in Plex's web-client/image config, not in +any repo service. + +**frank-archon bf16 no-spec cutover** + + +- `/scripts/frank-archon.env` — bf16 no-spec cutover, FINAL shape after the + night ladder: MAXLEN **241664** x SEQS 3, KV_BYTES **10,184,000,000**/rank, + KV_DTYPE auto (bf16), MNBT 1024, GATHER CHUNKS=32/LANES=64, UTIL 0.98; + FRANK_ARCHON_SPEC deleted (no spec). Engine measured at the winner: + "KV cache size: 732,359 tokens, Maximum concurrency for 241,664 tokens per + request: 3.03x". +- **The night ladder (Frank's method).** 262144 OOMed at CUDA graph capture + (`Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB + allocated by PyTorch) and 245760 OOMed identically; **237568 PASSED** + (KV 718,606 tokens) and the +4096 bonus step **241664 PASSED and is + PRODUCTION**. 241,664 is max possible on this box at util 0.98 — the + model's native 262,144 does not fit with a 3-seq contract. Every rung was + booted under a transient systemd unit at uid dank with + `LimitMEMLOCK=infinity` (the rootless bench probe masked the production + memory environment) with full engine-log capture. +- **Util stays 0.98.** The earlier 0.90 edit was wrong — lowering + `--gpu-memory-utilization` shrinks the engine's own budget rather than + buying slack; Frank forbade it and the winning ladder ran at 0.98. +- `/scripts/serve-frank-archon.sh` — `--speculative-config` line removed; + `--max-num-batched-tokens` and the `-e R4D_LRU_GATHER_CHUNKS/LANES` + container-env passthrough added; shape header rewritten. +- `/tests/frank_archon_unit.bats` — seam re-pinned: bf16-KV-present, + spec-absent-by-measurement, gather-32x64-present, MNBT-1024 tests; the + qualified-shape and flags tests flipped to the new env keys. +- Overnight redo campaign (prefetch OFF everywhere) tabled in + `/serverAppConfigs-docs/LLM/frank-archon.md` + with the 2026-10-01 cutover section: gather 32x64 +18.7%, MNBT flat + 1024-4096, KV pool law (7,926 fp8 / 14,047 bf16 B/token), arena flat zone, + six-slot refusal, bf16 3x262144 boot failure root cause (CUDA OOM at + KV_BYTES 11.3e9; boots clean at 11.047e9 — captured from journald `-o json`). +- `/AGENTS.md` — fleet table frank-archon row rewritten to the bf16 shape. +- `/serverAppConfigs-docs/LLM/index.md` — fleet intro + mermaid node + updated out of the GLM era. +- **Client contract fixed (thinking + tools).** The live endpoint returned a + null `reasoning` field and dumped the thinking text INLINE in `content` + with a dangling `` (measured: `'We need answer…Brief.\n\n\n391'`) + — the malformed thinking block clients choke on — because the deployed argv + had no `--reasoning-parser`. The GLM rollback (`d91379d`) had dropped all + three contract flags and the qwen rollback inherited the gap. Restored from + `ecaad36`'s values, confirmed against this fork's registries: + `--reasoning-parser qwen3`, `--enable-auto-tool-choice + --tool-call-parser qwen3_xml`, `--default-chat-template-kwargs + '{"enable_thinking": true, "preserve_thinking": true}'`. The seam's + "NO tool dialect" test is flipped to a PRESENT test. +- **Final shape (Frank's call): 5 seqs x the full native 262144 window at + fp8 KV.** 6 x 241664 at fp8 died (`Tried to allocate 10.70 GiB`, 9.18 GiB + free) — six full-length contexts do not fit at any KV dtype — so the + contract moved to five FULL windows: 5 x 262144 fp8 PASSED on the first + rung, KV_BYTES 10,388,766,720/rank, measured **1,350,182 tokens / + 5.15x concurrency** (39,462 tokens of margin). Seam re-pinned; the bf16 + three-context shape (241,664, KV 732,359 / 3.03x, measured earlier the + same night) is the documented max at 3 contexts. +- Harness `/home/blank/.omp/agent/models.yml` — contextWindow back to 262144 + with the five-window fp8 shape (compaction gap 183,501 + 36,864 = 220,365 + <= 262,144) and + `preserve_thinking` moved from a bare top-level body field to + `chat_template_kwargs` (the form vLLM actually reads). +- **Runtime-survivability correction: five slots -> FOUR.** 5 x 262144 fp8 + passed every boot gate (KV 1,350,182 tokens / 5.15x) but left 0 bytes free + at runtime; a real client request (995-token prompt, `max_tokens=261149`, + sampling params straight from models.yml) killed Worker_TP1 with + `torch.OutOfMemoryError: Tried to allocate 50.00 MiB ... 0 bytes free` and + the engine shut down. Boot is not proof of runtime survivability — after a + knife-edge shape change, send a large-`max_tokens` request before calling it + done. FOUR slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93 + GiB per rank for activations/graph capture. Frank's call 2026-10-01. +- `/scripts/frank-archon.service` — **Restart=always, not on-failure** (plus + `StartLimitIntervalSec=0` and `OOMScoreAdjust=-500`). A runtime worker fault + makes vLLM's API server exit CLEAN, so ExecStart returned 0, systemd logged + "Deactivated successfully", and on-failure NEVER fired: the endpoint stayed + down until a human noticed. An explicit `systemctl stop` still wins, so + bench windows and deploys are unaffected. +- **Final shape (Frank's call, 2026-10-01): 3 x 262144, pool at the FLOOR, + MNBT 2048, rest to the arena.** MAXLEN 262144 x SEQS 3 = 786,432 tokens of + contract; measured fp8 costs 7,698 B/token, so 6.054e9 B is the pool floor + and the deployed 6,100,000,000 B leaves a ~5.9k-token margin while handing + the remaining ~2.2 GiB (versus the 4-slot shape) to the expert arena — the + decode-critical cache the fork sizes from the remainder. MNBT 2048 costs the + arena nothing (the sweep measured 6,499 slots at every rung <= 4096). +- **DCP investigated and closed** (see the parallelism survey in the runbook): + `--decode-context-parallel-size 2` is rejected at boot — DCP for GQA requires + TP > the model's KV heads (2), i.e. TP >= 4, i.e. four GPUs. Not a tuning + knob on this box. +- Production bring-up: the shape boots and serves on :7770 — engine log + verified (KV cache size, concurrency, MNBT 1024, gather 32x64) plus live + contract probes (reasoning split, tool_calls). Rollback target for the + 2026-09-29 fp8+MTP4 shape: git tag `frank-archon-fp8-mtp-20261001`. + +**frank-dragoon cutover: gufo -> vLLM TheRock, MiMo-V2.6 BF16 + cross-model MTP** + + +- **Question that started it:** can a Qwen3.5 MTP drafter attach to + frank-zealot, and how does vLLM run the FULL BF16 MiMo checkpoint? Answer 1: + not on frank-zealot — SGLang reads the MTP head from the TARGET checkpoint + and MiMo ships ZERO `mtp.*` tensors (base and INT8 alike). Answer 2: the + digest-pinned `kyuz0/vllm-therock-gfx1151` image (vLLM 0.30 / torch 2.11 / + TheRock ROCm 10) boots BF16 MiMo on the gfx1151 iGPU in ~200-240 s, and + vLLM reads the draft from a SEPARATE path — so cross-model MTP from + `Qwen/Qwen3.5-9B` (the distill's source, identical 4096/248320/32-layer + geometry) works. Frank's call: ship that as frank-dragoon. +- **Bench window (frank-dragoon stopped, one session, temp 0, 28,647-token + prefill / 512 decode):** base 17,876 t/s prefill / 13.1 t/s decode (13.1 + is ~94% of the LPDDR5x BF16 roof - decode here is bandwidth, not a bug); + +DFlash-8 7,976 / 17.4 (acceptance 59%/26% per position); +MTP-2 7,380 / + 19.8 (71%/44%, token rate 57.4%). Deployed MTP-2: +51% decode. Both spec + arms pay ~55-59% of prefill. `num_speculative_tokens` beyond 2 unmeasured. +- **Measured numerics law:** spec arms do NOT reproduce the no-spec greedy + text (bf16 batched verification reorders reductions; near-tie argmaxes + cascade). vLLM spec decode is distribution-preserving, not + bitwise-greedy-preserving. +- **Client-contract delta wired:** vLLM emits `reasoning`, not + `reasoning_content` - `models.yml` frank-dragoon block now sets + `reasoningContentField: "reasoning"`, and the extraBody moved to the + Qwen3.5-family recipe (0.7/0.8) with `chat_template_kwargs` + (a top-level `preserve_thinking` is ignored by vLLM). +- Deployer gap found pre-deploy: the spec-draft gate demanded a single + `model.safetensors` (the dflash layout) and would have rejected the + deployed MTP arm (4 shards behind an index); fixed to accept both + layouts, gated by bats. `z-lab/Qwen3.5-9B-DFlash`'s "vLLM PR #40898 + unmerged" card claim is FALSE - `vllm/v1/spec_decode/dflash.py` ships in + 0.30 (verified in-image); the drafter stays on disk as the alternative. +- Host state: +38 GiB fetched (MiMo BF16 17.53 GiB, Qwen3.5-9B 18 GiB, + DFlash 2.58 GiB), disk down to ~26 GiB free - reclaim pass due. gufo + rollback pinned at git tag `frank-dragoon-gufo-q4kxl-20261001`; engine, + Q4_K_XL payload and DFlash2 drafter all remain on the host. +- First deploy attempt #2 crash-looped the container 51 times: the serve + script handed vLLM the HOST model path while the tree is mounted at + `/models` inside the container — transformers rejected it as a malformed + repo id. Fixed by converting to the container path (the spec draft was + already converted; the target was not), pinned by bats. Also fixed the + deployer's own draft gate (same single-file bug as the serve script's). +- Reclaim (Frank approved): Swift-1.5 Q8_K_XL payload (30 G, refetchable via + `scripts/download-frank-dragoon-swift.sh`) and the orphaned flux-2-klein + tree under `models/checkpoints` (23 G; ComfyUI mounts + `comfyui-data/models`, not this) purged via + `~/.AGENTTRASH/20261001-frank-dragoon-reclaim/` — 52 G freed. KEPT the + 1.1 G `Qwen3.8-27B-DFlash2-Q4_K_M.gguf` (it is the gufo rollback's + drafter, not a remnant). + ## 2026-09-30 - **frank-archon cut over to GLM-5.3-Flash W4A16 int4 — it serves, but NOT at the 35 tok/s target, and the shortfall is measured rather than inferred.** @@ -650,7 +1949,6 @@ - Rollback: commit `2339fae` (qwen FP8 + MTP4); the qwen payload and the R9V state dirs are intact on disk. - ## 2026-09-29 - **frank-dragoon tuning investigation: a measured NEGATIVE result, nothing changed.** Frank asked for iterative decode/prefill improvement at a 32k @@ -1166,1216 +2464,3 @@ - `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — Repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` stages conflicting drop-ins, comments out the stale sysctl.conf line, and applies `sysctl --system` (the boot code path). - `/tests/caddy_quadlet.bats` (new) + caddyfile/deployment_flow/pihole seams — quadlet boot persistence, self-healing, network, env, and sysctl coverage. - `/serverAppConfigs-docs/Infrastructure/Caddy.md`, `/serverAppConfigs-docs/Infrastructure/Pi-hole.md`, `/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md`, `/serverAppConfigs-docs/Dankclaw/Gitea.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — quadlet runtime facts, the 2026-08-26 incident, the sysctl-conflict failure mode, canonical Caddy ownership. - -## 2026-10-01 — frank-archon RESTORED to fp8 3 x 262144 (bf16 boot fails) - -bf16 (--kv-cache-dtype auto) at 2 x 262144 did not stay up, with OR without ---expert-precise-memory: - -- with precise: NRestarts 1->3, each attempt ~6.7 min then - `Deactivated successfully` (API server drains after an EngineCore fatal - error) — cause not captured, the serve script rm'd the container. -- without precise (default 7.60 GiB overhead, arena 14.3 GiB): NRestarts - climbing, container `Exited (1)`, and the journal finally showed the real - error: - `RuntimeError: Worker failed with error 'Constraints violated - (L['query_start_loc'].size()[0])! You marked L['query_start_loc'].size()[0] - as dynamic but your code specialized it to be a constant (3)'` - — a torch.compile/dynamo dynamic-shape constraint violation, NOT an OOM. - `~/.cache/vllm` holds only `modelinfos` (16K) and each `podman run` gets a - fresh container filesystem, so a stale compile cache is ruled out: the - conflict is produced by this configuration's own compile path. - -Differences from the last shape measured serving (fp8 3 x 262144, MNBT 2048, -KV 6.1e9 -> 792,069 tokens / 3.02x / arena 6,248 / 3.59 GiB free): KV dtype -(fp8 -> bf16) and SEQS (3 -> 2). Both are candidates for the dynamo trigger; -`--enforce-eager` is the bypass that would sidestep the constraint path. - -Restored: fp8 3 x 262144, MNBT 2048, KV_BYTES 6,100,000,000, no precise flag. -Next: bisect bf16 vs SEQS in a window (bf16 + SEQS=3 isolates the dtype). - -## 2026-10-01 — frank-archon: --expert-precise-memory + bf16 = boot crash loop - -**Reverted.** The two changes Frank asked for were applied together and the -result crash-looped: `NRestarts` climbed 1 -> 2 -> 3, each attempt running -~6.7 minutes then logging `Deactivated successfully` (vLLM's API server drains -and exits 0 after an EngineCore fatal error, so `podman wait` returns 0; the -new `Restart=always` turned a silent death into a visible loop). - -The budget arithmetic explains it without needing the error text, because the -arena is DEVICE-resident (the fp8 3-window shape's plan predicts 28.27 GiB and -that is exactly what `rocm-smi` measured): - -| line | bf16 2 x 262144 + precise | fp8 3 x 262144 | -|---|---|---| -| non-expert weights | 6.28 | 6.28 | -| pinned pool | 6.80 | 5.68 | -| CUDA graphs | 0.55 | 0.55 | -| arena | 16.65 (precise: overhead 5.25) | 14.31 (default: overhead 7.60) | -| non-torch | ~1.45 | ~1.45 | -| **total** | **31.73 of 31.86 GiB** | 28.27 (measured) | -| free at boot | **~0.13 GiB** | 3.59 | - -`--expert-precise-memory` hands the 2.35 GiB compile/graph-capture headroom to -the arena, and bf16 simultaneously adds 1.08 GiB of pinned pool; together they -put graph capture ~0.13 GiB from the ceiling, where every transient allocation -dies. So the flag is dropped and bf16 2 x 262144 ships with the default -overhead: arena 14.31 GiB, ~2.4 GiB free at boot. - -**Lesson:** the registry's "compile headroom" is not slack to be reclaimed by -default — it is the budget for the phase that allocates last. Reclaim it only -by paying for it elsewhere (smaller pool, fewer sessions, or a shorter context). - -## 2026-10-01 — frank-archon: 2 bf16 windows + --expert-precise-memory - -Frank's final shape for the night. Two changes at once, both measured in -principle before the boot: - -- **KV is bf16 (`--kv-cache-dtype auto`), two full-size windows (2 × 262144).** - bf16 measures **13,839 B/token** (from the bf16 3×241664 boot: - 11,047,010,304 B -> 798,280 tokens), decomposing as **12,276 attention + - 1,563 mamba/side caches** against fp8's **6,138 + 1,563 = 7,701**. So - 524,288 tokens needs 7.256e9 B; pinned at 7,300,000,000 (~527,500 tokens, - 3,206 margin). Cost: the pool goes 5.68 -> 6.76 GiB per rank. -- **`--expert-precise-memory`.** The registry's default overhead for this - architecture is 7.60 GiB = the fitted 5.25 GiB (gfx1201 TP2 - nbt4096/seqs16/len262144) + 2.35 GiB of compile/graph-capture headroom. - The flag selects the fitted figure, which moves only the plan's `room` term - — arena 6,248 -> ~7,278 slots (~1,025 more, at 2.344 MiB/slot) — at the - cost of boot-time capture slack. The overhead formula adds 0.22 GiB per - 1024 nbt above 4096, so MNBT 2048 pays nothing. - -Budget note worth keeping: the arena is `(budget - non_expert_weights - -kv_reserve - graphs - overhead) // expert_bytes`, where `kv_reserve` is the -planner's OWN one-max-model-len reserve (2.48 GiB at TP2) and does NOT scale -with SEQS. The pinned `--kv-cache-memory-bytes` therefore does not shrink the -planned arena; it comes out of free VRAM instead. - -## 2026-10-01 — frank-dragoon: Q4_K_XL payload + 4 x 163840 sessions - -Frank's call. Payload swapped from ajgazin's **abliterated** Swift-1.5 -UD-Q8_K_XL (29.3 GiB) to **unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL** (16.35 GiB, -sha256 3f227079…b01e, vanilla/Apache-2.0 — the endpoint is no longer -uncensored). New fetcher `scripts/download-frank-dragoon-q4.sh`; the DFlash2 -drafter and its fetcher are unchanged. - -Shape: **`--sessions 4 --context 163840`** with unquantized KV. gufo has no -KV-quantization lever at all (`execution_policy.hpp` pins `kv_cache_storage` -to `kFp16`), so "unquantized" is inherent. - -KV arithmetic (why this fits where one 262144 session used to sit): -f16 KV costs **64 KiB/token** — 16 full-attention layers (the hybrid is -`full_attention_interval: 4` over 64 layers) x 4 KV heads x 256 head_dim x -2 (K,V) x 2 bytes; the other 48 layers are linear-attention with a -per-SEQUENCE fp32 state (~144 MiB), not a per-token cache. 4 x 163840 = -655,360 tokens = 40.0 GiB of KV, and each session is exactly 10 GiB. Budget: -weights 16.35 + KV 40.0 + drafter 1.07 + linear state 0.56 + runtime ~4 = -~62 GiB against the 114 GiB GTT. - -## 2026-10-01 — e2e gate flakiness (12-crawl) — do not panic at the banner - -`specs/12-crawl.spec.ts:23` ("every user open link reaches its app") failed the -deploy gate twice and then passed cleanly on a manual re-run: -- run 1 (deploy): `Expected "plex.franksdank.cloud"` / `Received "app.plex.tv"` - — Plex redirects its web UI to the hosted app; the same failure appeared on - the previous deploy too. -- run 2 (manual): `page.goto: net::ERR_ABORTED at https://www.franksdank.cloud/` - — the homepage answered `http=200` in 40 ms from the dev box at the same - moment, so this is a browser-level flake. -- run 3 (manual): **33 passed**, exit 0. - -So a red gate on this spec alone is NOT proof the fleet is broken, and the -deployer's `DEPLOY INCOMPLETE` banner can be raised by a flake. Re-run -`tests/e2e/run.sh` before treating it as real; the Plex redirect is the one -stable underlying issue and it lives in Plex's web-client/image config, not in -any repo service. - -## 2026-10-01 — frank-archon bf16 no-spec cutover - -- `/scripts/frank-archon.env` — bf16 no-spec cutover, FINAL shape after the - night ladder: MAXLEN **241664** x SEQS 3, KV_BYTES **10,184,000,000**/rank, - KV_DTYPE auto (bf16), MNBT 1024, GATHER CHUNKS=32/LANES=64, UTIL 0.98; - FRANK_ARCHON_SPEC deleted (no spec). Engine measured at the winner: - "KV cache size: 732,359 tokens, Maximum concurrency for 241,664 tokens per - request: 3.03x". -- **The night ladder (Frank's method).** 262144 OOMed at CUDA graph capture - (`Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB - allocated by PyTorch) and 245760 OOMed identically; **237568 PASSED** - (KV 718,606 tokens) and the +4096 bonus step **241664 PASSED and is - PRODUCTION**. 241,664 is max possible on this box at util 0.98 — the - model's native 262,144 does not fit with a 3-seq contract. Every rung was - booted under a transient systemd unit at uid dank with - `LimitMEMLOCK=infinity` (the rootless bench probe masked the production - memory environment) with full engine-log capture. -- **Util stays 0.98.** The earlier 0.90 edit was wrong — lowering - `--gpu-memory-utilization` shrinks the engine's own budget rather than - buying slack; Frank forbade it and the winning ladder ran at 0.98. -- `/scripts/serve-frank-archon.sh` — `--speculative-config` line removed; - `--max-num-batched-tokens` and the `-e R4D_LRU_GATHER_CHUNKS/LANES` - container-env passthrough added; shape header rewritten. -- `/tests/frank_archon_unit.bats` — seam re-pinned: bf16-KV-present, - spec-absent-by-measurement, gather-32x64-present, MNBT-1024 tests; the - qualified-shape and flags tests flipped to the new env keys. -- Overnight redo campaign (prefetch OFF everywhere) tabled in - `/serverAppConfigs-docs/LLM/frank-archon.md` - with the 2026-10-01 cutover section: gather 32x64 +18.7%, MNBT flat - 1024-4096, KV pool law (7,926 fp8 / 14,047 bf16 B/token), arena flat zone, - six-slot refusal, bf16 3x262144 boot failure root cause (CUDA OOM at - KV_BYTES 11.3e9; boots clean at 11.047e9 — captured from journald `-o json`). -- `/AGENTS.md` — fleet table frank-archon row rewritten to the bf16 shape. -- `/serverAppConfigs-docs/LLM/index.md` — fleet intro + mermaid node - updated out of the GLM era. -- **Client contract fixed (thinking + tools).** The live endpoint returned a - null `reasoning` field and dumped the thinking text INLINE in `content` - with a dangling `` (measured: `'We need answer…Brief.\n\n\n391'`) - — the malformed thinking block clients choke on — because the deployed argv - had no `--reasoning-parser`. The GLM rollback (`d91379d`) had dropped all - three contract flags and the qwen rollback inherited the gap. Restored from - `ecaad36`'s values, confirmed against this fork's registries: - `--reasoning-parser qwen3`, `--enable-auto-tool-choice - --tool-call-parser qwen3_xml`, `--default-chat-template-kwargs - '{"enable_thinking": true, "preserve_thinking": true}'`. The seam's - "NO tool dialect" test is flipped to a PRESENT test. -- **Final shape (Frank's call): 5 seqs x the full native 262144 window at - fp8 KV.** 6 x 241664 at fp8 died (`Tried to allocate 10.70 GiB`, 9.18 GiB - free) — six full-length contexts do not fit at any KV dtype — so the - contract moved to five FULL windows: 5 x 262144 fp8 PASSED on the first - rung, KV_BYTES 10,388,766,720/rank, measured **1,350,182 tokens / - 5.15x concurrency** (39,462 tokens of margin). Seam re-pinned; the bf16 - three-context shape (241,664, KV 732,359 / 3.03x, measured earlier the - same night) is the documented max at 3 contexts. -- Harness `/home/blank/.omp/agent/models.yml` — contextWindow back to 262144 - with the five-window fp8 shape (compaction gap 183,501 + 36,864 = 220,365 - <= 262,144) and - `preserve_thinking` moved from a bare top-level body field to - `chat_template_kwargs` (the form vLLM actually reads). -- **Runtime-survivability correction: five slots -> FOUR.** 5 x 262144 fp8 - passed every boot gate (KV 1,350,182 tokens / 5.15x) but left 0 bytes free - at runtime; a real client request (995-token prompt, `max_tokens=261149`, - sampling params straight from models.yml) killed Worker_TP1 with - `torch.OutOfMemoryError: Tried to allocate 50.00 MiB ... 0 bytes free` and - the engine shut down. Boot is not proof of runtime survivability — after a - knife-edge shape change, send a large-`max_tokens` request before calling it - done. FOUR slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93 - GiB per rank for activations/graph capture. Frank's call 2026-10-01. -- `/scripts/frank-archon.service` — **Restart=always, not on-failure** (plus - `StartLimitIntervalSec=0` and `OOMScoreAdjust=-500`). A runtime worker fault - makes vLLM's API server exit CLEAN, so ExecStart returned 0, systemd logged - "Deactivated successfully", and on-failure NEVER fired: the endpoint stayed - down until a human noticed. An explicit `systemctl stop` still wins, so - bench windows and deploys are unaffected. -- **Final shape (Frank's call, 2026-10-01): 3 x 262144, pool at the FLOOR, - MNBT 2048, rest to the arena.** MAXLEN 262144 x SEQS 3 = 786,432 tokens of - contract; measured fp8 costs 7,698 B/token, so 6.054e9 B is the pool floor - and the deployed 6,100,000,000 B leaves a ~5.9k-token margin while handing - the remaining ~2.2 GiB (versus the 4-slot shape) to the expert arena — the - decode-critical cache the fork sizes from the remainder. MNBT 2048 costs the - arena nothing (the sweep measured 6,499 slots at every rung <= 4096). -- **DCP investigated and closed** (see the parallelism survey in the runbook): - `--decode-context-parallel-size 2` is rejected at boot — DCP for GQA requires - TP > the model's KV heads (2), i.e. TP >= 4, i.e. four GPUs. Not a tuning - knob on this box. -- Production bring-up: the shape boots and serves on :7770 — engine log - verified (KV cache size, concurrency, MNBT 1024, gather 32x64) plus live - contract probes (reasoning split, tool_calls). Rollback target for the - 2026-09-29 fp8+MTP4 shape: git tag `frank-archon-fp8-mtp-20261001`. - -## 2026-10-01 — frank-dragoon cutover: gufo -> vLLM TheRock, MiMo-V2.6 BF16 + cross-model MTP - -- **Question that started it:** can a Qwen3.5 MTP drafter attach to - frank-zealot, and how does vLLM run the FULL BF16 MiMo checkpoint? Answer 1: - not on frank-zealot — SGLang reads the MTP head from the TARGET checkpoint - and MiMo ships ZERO `mtp.*` tensors (base and INT8 alike). Answer 2: the - digest-pinned `kyuz0/vllm-therock-gfx1151` image (vLLM 0.30 / torch 2.11 / - TheRock ROCm 10) boots BF16 MiMo on the gfx1151 iGPU in ~200-240 s, and - vLLM reads the draft from a SEPARATE path — so cross-model MTP from - `Qwen/Qwen3.5-9B` (the distill's source, identical 4096/248320/32-layer - geometry) works. Frank's call: ship that as frank-dragoon. -- **Bench window (frank-dragoon stopped, one session, temp 0, 28,647-token - prefill / 512 decode):** base 17,876 t/s prefill / 13.1 t/s decode (13.1 - is ~94% of the LPDDR5x BF16 roof - decode here is bandwidth, not a bug); - +DFlash-8 7,976 / 17.4 (acceptance 59%/26% per position); +MTP-2 7,380 / - 19.8 (71%/44%, token rate 57.4%). Deployed MTP-2: +51% decode. Both spec - arms pay ~55-59% of prefill. `num_speculative_tokens` beyond 2 unmeasured. -- **Measured numerics law:** spec arms do NOT reproduce the no-spec greedy - text (bf16 batched verification reorders reductions; near-tie argmaxes - cascade). vLLM spec decode is distribution-preserving, not - bitwise-greedy-preserving. -- **Client-contract delta wired:** vLLM emits `reasoning`, not - `reasoning_content` - `models.yml` frank-dragoon block now sets - `reasoningContentField: "reasoning"`, and the extraBody moved to the - Qwen3.5-family recipe (0.7/0.8) with `chat_template_kwargs` - (a top-level `preserve_thinking` is ignored by vLLM). -- Deployer gap found pre-deploy: the spec-draft gate demanded a single - `model.safetensors` (the dflash layout) and would have rejected the - deployed MTP arm (4 shards behind an index); fixed to accept both - layouts, gated by bats. `z-lab/Qwen3.5-9B-DFlash`'s "vLLM PR #40898 - unmerged" card claim is FALSE - `vllm/v1/spec_decode/dflash.py` ships in - 0.30 (verified in-image); the drafter stays on disk as the alternative. -- Host state: +38 GiB fetched (MiMo BF16 17.53 GiB, Qwen3.5-9B 18 GiB, - DFlash 2.58 GiB), disk down to ~26 GiB free - reclaim pass due. gufo - rollback pinned at git tag `frank-dragoon-gufo-q4kxl-20261001`; engine, - Q4_K_XL payload and DFlash2 drafter all remain on the host. -- First deploy attempt #2 crash-looped the container 51 times: the serve - script handed vLLM the HOST model path while the tree is mounted at - `/models` inside the container — transformers rejected it as a malformed - repo id. Fixed by converting to the container path (the spec draft was - already converted; the target was not), pinned by bats. Also fixed the - deployer's own draft gate (same single-file bug as the serve script's). -- Reclaim (Frank approved): Swift-1.5 Q8_K_XL payload (30 G, refetchable via - `scripts/download-frank-dragoon-swift.sh`) and the orphaned flux-2-klein - tree under `models/checkpoints` (23 G; ComfyUI mounts - `comfyui-data/models`, not this) purged via - `~/.AGENTTRASH/20261001-frank-dragoon-reclaim/` — 52 G freed. KEPT the - 1.1 G `Qwen3.8-27B-DFlash2-Q4_K_M.gguf` (it is the gufo rollback's - drafter, not a remnant). - -## 2026-10-02 — frank-dragoon cutover: vLLM MiMo -> ROCmFPX Q8_0_ROCMFPX_AGENT + MTP - -- 2026-10-02 — frank-dragoon: vLLM MiMo → ROCmFPX `Q8_0_ROCMFPX_AGENT` - payload + MTP from the repo's own bundled head, one session at the native - 262,144. See [[frank-dragoon]]. -- The runtime is **forced by the file**: tensor type 103 - (`GGML_TYPE_Q8_0_ROCMFPX`, ftype 115) is unreadable by stock llama.cpp, - gufo and every other loader on the fleet — only the kingjones30/ROCmFPX - fork can read it, so the engine is pinned to commit - `36e9acd40e10a87cd3c3ef8ec734668757dc8520` and built by - `scripts/build-rocmfpx.sh`. The AGENT routing variant was chosen over the - card's plain Q8_0 because it keeps 187 output-head tensors at plain Q8_0 - instead of 2, which measured better MTP acceptance. MTP was the whole - performance story of the shape (roughly +120 % decode, ladder on the - runbook). -- Rollback: git tag `frank-dragoon-gufo-q4kxl-20261001`; the retired - vLLM-era fetchers moved to `old/`. Runbook [[frank-dragoon]]. - -## 2026-10-02 — frank-zealot: Q5_K_M, then three contexts + vision, sized by two boots - -- 2026-10-02 — frank-zealot: Q5_K_M payload, then three contexts + vision - in one day, sized by two boots (`CTX_SIZE=540672 PARALLEL=3`, f16 K + - q8_0 V under flash attention, `--mmproj`). See [[frank-zealot]]. -- **The pool came from two boots, not a formula.** Differencing the boots - prices the non-KV floor — and prices the vision tower at **~1.4 GiB for - a 918 MB file**: the encoder's own buffers ride with it. The first 3-slot - deploy booted healthy but left less than the ordered 2–3 GiB band free, - which is why the seam now prices non-KV at the measured value and - REJECTS that shape. -- Three contexts cost window, and that is the tradeoff Frank was told: - 180,224 tokens per conversation, not the native 262,144; a longer prompt - is a hard `400 exceed_context_size_error`, so `models.yml` follows the - slot, not the card. -- **Vision contract gotcha:** `/v1/chat/completions` ignores the legacy - top-level `images` field — that request answered "Brown." for the same - red image, i.e. it generated from text alone. Only the `image_url` - content part reaches the model. `/completion` accepts `images` but this - build still requires `messages`. -- Deployer gates on the projector too (`LLM_SMALL_MMPROJ`), so a missing - mmproj cannot silently ship a blind endpoint; `MMPROJ_FILE=` - (explicitly empty) in the env file is the documented text-only opt-out - and returns the tower's ~1.4 GiB to KV. -- Reclaim staged, **not purged, awaiting approval**: - `~/.AGENTTRASH/20261002-retired-llm-payloads/` of superseded payloads. - Runbook [[frank-zealot]]. - -## 2026-10-02 — frank-archon shape D: 3 x 200704, fp8, pool sized for 2 contexts - -Frank's order: shrink frank-archon to a 200704-token window with about two -resident full-length contexts, cut the KV pool, and drop MNBT to 1024. The -first attempt (`30904ba`, 2 x 200704 + SEQS=2) **crash-looped**; the revert -went live as `e460bbb`, and the shape was then re-applied one law at a time on -top of the proven-good base, root cause first. - -- **ROOT CAUSE: `--max-num-seqs 2` cannot compile this model. It was never - OOM, never MNBT, never the KV dtype.** Four single-variable boot probes - against the then-deployed shape (production stopped for each, candidate - booted AS the service): 262144 x SEQS 2 fails at BOTH MNBT rungs with - `torch.fx.experimental.symbolic_shapes.ConstraintViolationError: - Constraints violated (L['query_start_loc'].size()[0]) ... specialized it - to be a constant (3)`, raised from AOT `build_guards(strict_error=True)` - during `determine_available_memory -> profile_run -> _dummy_run`, ~7 min - in; SEQS 3 is healthy at both rungs. `model_runner.py` sizes that input - buffer `max_num_reqs + 1` and `_dummy_run` sets `num_reqs = - min(max_num_batched_tokens, max_num_reqs)`, so its width is **SEQS + 1** - for ANY `MNBT >= SEQS` — 3 at SEQS 2, exactly the constant in the - message; MNBT only moves the compile RANGE, which is why the identical - error appears at both rungs. `--enforce-eager` bypasses the path but - drops CUDA graphs. The 2026-10-01 bf16 2 x 262144 failure was this same - error class; its message named "KV dtype and SEQS" together and nobody - bisected, so the dtype was blamed — it was a red herring. -- **THE ARENA LEVER IS `max_model_len`, NOT `--kv-cache-memory-bytes`.** - The planner sizes expert cache as `budget - non-expert - KV reserve - - graphs - overhead`, and the KV reserve is registry-priced at **9190 - B/token x max_model_len**, independent of the pool override: d(reserve) - equalled d(arena) exactly, and the multi-GB pool cut bought **zero** - arena slots. The previous session's premise — "the pool cut becomes - arena, ~7,500 slots" — was wrong; the pool cut is a runtime-HEADROOM - lever (the shape it replaced died with `0 bytes free` on a real - 995-token request). -- **Final shape (option D):** the argv keeps the only slot count that - compiles; the POOL caps resident conversations (ledger on - [[frank-archon]]). Measured live, one prediction corrected: the arena - prediction landed exactly, the concurrency one did not — **the per-token - KV price is shape-dependent** (7,701 B/token at MAXLEN 262144 vs **7,892** - at 200704, ~2.5 % dearer). Never extrapolate a concurrency figure across - geometries. Smoke probes confirmed the 3 slots real, native `tool_calls`, - and that `reasoning_effort` accepts only {xhigh, medium, low}, answering - 400 for anything else. -- **Harness contract followed the window:** `models.yml` `contextWindow` - 262144 -> **200704**; side effect: frank-archon's compaction request fits - inside a frank-zealot slot again. A comment-only follow-up `295d578` was - deployed so the per-service revision marker sits at HEAD. Rollback for - the pre-shape-D fp8+MTP4 era stays git tag - `frank-archon-fp8-mtp-20261001`. Runbook [[frank-archon]]. - -## 2026-10-02 - plex: the readiness gate probed HTTP against a TLS-only service - -Found by accident, and it had been latent since the secure-connections posture -was set: `deploy_plex` gated on `curl -fsS http://127.0.0.1:32400/identity`, -which **cannot** succeed. PMS drops plain HTTP off-loopback - the probe's -connection arrives on the container's eth0 - so Plex accepts the TCP connection -and then answers with nothing, and curl reports "Empty reply from server" (52). - -- Trigger: a bare `./scripts/deploy.sh dankserver` (no `--service`) drains - the whole pending backlog, not just the service you were thinking about. - plex's per-service revision had been behind long enough that this gate - had never run against the current posture; it burned its full 600 s and - failed the run, and every service after plex in the order was left - undeployed. -- **Plex was healthy the entire time.** `https://127.0.0.1:32400/identity` - returned 200 with the MediaContainer XML in 44 ms while the gate was - still timing out; the in-container plain-HTTP path also answers 200 — - loopback is the one path PMS serves in the clear, which is exactly why - the host-side probe could not see the difference. -- **The repo already knew this law, in a different file:** - `tests/caddyfile.bats` pins the plex vhosts to a TLS upstream with the - comment "PMS drops plain HTTP off-loopback". The deployer's gate never - got the same treatment. Fixed to `curl -fsSk --max-time 3 - https://127.0.0.1:32400/identity` (`-k`: PMS serves its own self-signed - cert on 32400). Re-deployed `39f0e0d`: plex recorded deployed, e2e - 33/33, frank-archon untouched (`health=200`, NRestarts=0). -- **Operating lesson: `deploy.sh ` with no `--service` is a - fleet-wide drain.** It restarts every service whose recorded revision is - behind HEAD and whose paths changed since. To touch one service, name it. - -## 2026-10-02 — frank-archon shape E: `--expert-precise-memory` (+1,026 arena slots), a mounted compile cache, and the P2P answer - -Shape D left one question open — the plan's overhead term was reserving 2.35 GiB -of compile headroom the boot never used — and the answer to Frank's free-VRAM -question arrived together with a boot-cost bug that had been costing 181 s on -every restart since the cutover. - -- **`--expert-precise-memory` is real and it fits (option A).** Boot-probed on - the shape-D base with production stopped, the candidate booted AS the service - (`FRANK_ARCHON_ENV_FILE=/tmp/probe-b.env bash scripts/serve-frank-archon.sh`, - trap-guaranteed restore), the numbers read off the engine's own plan lines: - - | plan line | shape D | shape E | - |---|---|---| - | `runtime overhead (non-torch + buffers + activation)` | 7.60 GiB | **5.25 GiB** (`precise (gfx1201 TP2 nbt4096/seqs16/len262144)`) | - | `expert cache space` | 14.87 GiB | **17.22 GiB** | - | slots | 6,496 | **7,522** (+1,026, +16 %) | - | `swap region` | — | 13.7 GiB / 5,986 slots, 133-134 per layer | - | boot | — | **healthy in 420 s**, no OOM | - - The flag also prints `'expert_precise_memory': True` in `non-default args`, so - a silent argparse drop cannot hide. It was refused on 2026-10-01 only because - the 2 x 262144 bf16 shape was already 0.13 GiB under the ceiling — there the - flag was the last stack over the line, not the cause. -- **The pool was NOT cut (option A) because the pool is not an arena lever.** - A 2.2e9 arm was probed in the same window and reports `277,686 tokens, - 1.38x` — measured, rejected as the shipped value, kept as the conservative - fallback. With 3.15e9/rank the endpoint keeps shape D's 1.99x concurrency and - gains the arena, which is the shape D follow-up had actually been looking for. -- **THE COMPILE CACHE WAS BEING THROWN AWAY ON EVERY RESTART — 181 s each.** - Measured: a fresh container has **no `/root/.cache` at all** (`No such file or - directory`); after our boot it holds `/root/.cache/vllm` at - **1,154,714,333 B in 4,665 files** plus `/root/.cache/comgr` at 229 MB, inside - the ephemeral layer, and `podman rm -f` (every deploy, restart, reboot) - discards them. The engine names the loss itself: `Compiling a graph for compile - range (1, 1024) takes 173.19 s` ... `torch.compile took 181.20 s in total` ... - `init engine (profile, create kv cache, warmup model) took 217.26 s - (compilation: 181.20 s)`. The serve script now mounts - `FRANK_ARCHON_CACHE_DIR` at `/root/.cache` — the **parent**, so the vLLM - AOT/torch_compile tree (`VLLM_CACHE_ROOT`, default `/root/.cache/vllm`) and - the comgr JIT cache both persist, and nothing is hidden because the image has - no such directory. Same class of fix as the recipe's - `/cache/{triton,vllm,inductor}` volumes; the compile RANGE is part of the - cache key, so an MNBT change costs one recompilation, not a permanent tax. -- **`GPU_MAX_HW_QUEUES=2`** (HSA hardware queues per GPU) added at Frank's - direction. Verified a real knob rather than a no-op — the literal is read from - `/opt/rocm/core-10.0/lib/libhsa-runtime64.so` — and shipped deliberately - alongside the 32x64 gather geometry. **Not A/B measured.** -- **P2P: the hardware supports it, and the engine is already using it.** The - live log shows `r4d all-reduce INSTALLED (rank=0 max=49152KB finegrained=T…)` - on BOTH ranks and `Using ['R4D_AR', 'PYNCCL'] all-reduce backends` — R4D's - P2P path, with RCCL as fallback, measured at **45.274 GB/s bidirectional**. - Root ports `00:01.1`/`00:01.3` both report ACS `ReqRedir+`, so P2P hairpins at - the root complex; there is no XGMI. The old FAIL predates the 2026-09-20 board - swap. Kernel support confirmed: `CONFIG_HSA_AMD_P2P=y`, `CONFIG_PCI_P2PDMA=y`, - `CONFIG_HSA_AMD=y`, `CONFIG_ZONE_DEVICE=y`, `iommu=pt` on the cmdline. -- **Recipe diff (Frank's 3 x R9700 compose vs this podman run), ranked:** - - *Acted on:* persistent compile caches (above — the one real operational - gap), and `GPU_MAX_HW_QUEUES`. - - *Retracted:* pinned `--compilation-config cudagraph_capture_sizes` was on my - list; measured, it is **not** a gap here — capture is 3 PIECEWISE + 2 FULL - graphs, `Graph capturing finished in 3 secs, took 0.31 GiB` (0.11 GiB on the - lighter rank). No 15-52-size ladder to cut. - - *Not acted on — measured candidates, none shipped (no bench window; these - are CPU-side and our decode is gather-bound, so a spin-wait stealing cores - from the gather threads is plausible but unproven):* `OMP_NUM_THREADS=8` - (defaults to all 16 logical CPUs on this 7700X), `HSA_ENABLE_INTERRUPT=1`, - `HSA_ENABLE_MWAITX=1`, `HSA_ENABLE_SDMA`, `HSA_NO_SCRATCH_RECLAIM`, - `HSA_ENABLE_IPC_MODE_LEGACY`. All six HSA names verified real — the literals - are read out of `/opt/rocm/core-10.0/lib/libhsa-runtime64.so` — so they are - knobs, not no-ops; any of them needs its own A/B before shipping. - - *Deliberate divergence, keep ours:* no MTP (the recipe has it; measured - fastest here), `qwen3_xml` vs `qwen3_coder` (ours verified live), no - `--override-generation-config` (server-side sampling defaults), published - port + Caddy rather than `network_mode: host`. - - *Non-gaps:* `shm_size 32gb` (ours is `--ipc=host`), `memlock -1` (same), - `SYS_PTRACE` (same), `/dev/kfd` + `/dev/dri` (same), `--enable-chunked-prefill` - (V1 default), `plecache` volume (the flag is not even passed), and - `ROCR_VISIBLE_DEVICES=1,2,3` vs our `HIP_VISIBLE_DEVICES=0,1` — verified - equivalent here: `rocm-smi --showid` puts the two R9700s at GPU[0]/GPU[1] and - the 7700X iGPU at GPU[2], so our indices are the right two cards. -- **Harness note, because it cost two probe attempts:** a probe that reported an - instantaneous candidate death was a logging artifact of the inline SSH script, - not a crash — the container and its serve script must be polled by recorded id - and their logs dumped on the exit path, or a failure looks like a bad flag. - The flag was fine both times. -- Live: `health=200`, `NRestarts=0`, arena 17.22 GiB / 7,522 slots, KV 398,658 - tokens / 1.99x. Docs: [[frank-archon]]. -- **Deployed `1ac4d48`, e2e 33/33, and the live plan matched the probe exactly:** - `'expert_precise_memory': True`, `runtime overhead … 5.25 GiB`, `expert cache - space 17.22 GiB / 7522 slots`, `swap region 13.7 GiB / 5986 slots`, `GPU KV - cache size: 398,658 tokens … 1.99x`, pool 3,150,000,000, MNBT 1024. The - precise overhead and the arena growth are now production numbers, not a - probe's projection. -- **The compile-cache fix was then proven on the same endpoint, two consecutive - boots of the deployed shape:** cold `Compiling a graph for compile range - (1, 1024) takes 179.24 s` → `torch.compile took 188.26 s`; warm `Directly load - the compiled graph(s) for compile range (1, 1024) from the cache, took - 2.143 s` → `torch.compile took 3.97 s in total` → `init engine (profile, - create kv cache, warmup model) took 41.13 s (compilation: 3.98 s)`. The host - dir held 1.4 GB in 4,707 files after the cold boot. **Time to `/health` is - dominated by the 172.8 GiB payload load** (the warm boot still needed 490 s), - so the honest claim is ~184 s of inductor/AOT build recovered per restart — - not a proportional boot cut. - -## 2026-10-11 — frank-archon: unquantized (bf16) KV at 2 × 200,000 - -- 2026-10-11 — frank-archon: **unquantized (bf16) KV at 2 × 200,000** on - exact attention state, with pool, slab and host pool rebalanced around - the bigger cache. See [[frank-archon]]. -- Engine door: `--kv-cache-dtype` takes ONLY `bf16|fp8` — one dtype for the - whole cache, so llama.cpp's exact-K / quantized-V pair (what frank-zealot - runs) has **no radiance equivalent**. -- `--max-num-seqs 3 → 2` booted clean: the fleet's "seqs 2 cannot boot" law - is the retired tcclaviger vLLM fork's torch.compile `query_start_loc` - constraint, not radiance's — radiance booted seqs 2 in its fp8-native era - and the sweep had 2 and 3 tied inside noise. -- Harness law re-learned: radiance puts the thinking block in - `reasoning_content`, so a probe that reads `content` on a small budget - sees empty text with `finish_reason: stop` — a healthy server scored 0/2 - until the probe set `chat_template_kwargs.enable_thinking=false`. Tool - committed as `scripts/probe-frank-archon-kv.py`. -- The fp8-vs-bf16 control arms and the open keep-bf16-or-take-cheap-fp8 - call are recorded on [[frank-archon]]; the 2026-10-13 section below (in - file order above) closes that call in favour of bf16. - -## 2026-10-15 — frank-archon: publisher iq4r payload swap (MODEL_FILE-only) - -- 2026-10-15 — frank-archon: publisher iq4r payload swap, `MODEL_FILE`-only - — `StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe`, the engine author's - published rad-convert of the OFFICIAL Qwen3.8-Flash-Next, **no other - argument changes**, mindful the operator session runs on the endpoint - being swapped. See [[frank-archon]]. -- Disk: Frank authorized one `rm` — the spent BF16 source and the - two-era-old fp8-native `.rad` left `/home/dank/models`; fp8-native is now - rebuild-only (fetch + build scripts); rollback for THIS swap is the - previous `.rad`, on disk, by reverting the commit. -- Fetch: new `scripts/download-frank-archon-iq4r.sh` (hfget, exact bytes + - Hub LFS sha256, `--verify-only` default) as a transient unit. The cutover - **self-severed by design** — a detached `deploy.sh` survived the session - death when frank-archon restarted; engine healthy in ~100 s, and the - prefix tier RECOVERED across the swap (arch+tokenizer unchanged). -- e2e gate went red on three dankclaw specs — NOT the swap: dankclaw - rebooted 01:41 and the gluetun boot-race dependency-failed `searxng`, - `marreta`, and `marreta-browser` (jobs consumed while gluetun's start job - retried; `Restart=always` never armed because the services never ran). - Cascade restarted; skill codified - (`gluetun-dependent-boot-race-recovery`). frank-archon's own health gate - passed every deploy. - -## 2026-10-15 — authentik: two more human accounts (Dankclaw) - -- **Live — mitch (pk 22) and eric (pk 23) provisioned via `ak shell` ORM, - no repo objects**, each `@franksdank.cloud` in the human-user - template shape (password via `set_password` on stdin only — never argv, - so nothing lands in `ps`), verified through the exact PasswordStage code - path; zero `proxysession` rows for either, so no claim purge was due. -- `/serverAppConfigs-docs/Infrastructure/Authentik.md` — human-user census - extended with pk 22–23. - -## 2026-10-17 — frank-zealot: Bonsai-2-27B PQ2_0 trial + cutover (2 × 200,704, q8_0 KV) - -- 2026-10-17 — frank-zealot: **Ternary-Bonsai-2-27B (PQ2_0) cutover**, two - slots at 200,704 over q8_0 K + q8_0 V on the RTX 3090, fetched by - `scripts/download-frank-zealot-bonsai2.sh` (exact bytes + sha256, - `--verify-only`). See [[frank-zealot]]. -- Runtime door: the file is `file_type 141` with 402 tensors of custom GGML - type **142** plus `prism.hadamard.*` metadata — stock llama.cpp refuses - it, so the PrismML vendor build is **forced, not chosen**; the vendor - build serves q8_0 V under flash attention, so the source rebuild with - `GGML_CUDA_FA_ALL_QUANTS=ON` was cancelled as unnecessary. -- Trial-window law cost one boot: the scratch env invented - `KV_CACHE_TYPE` where the serve script reads - `CACHE_TYPE_K`/`CACHE_TYPE_V`, so the script's `:=f16` default silently - won and every arm booted f16. A silently-defaulted knob turns a whole - trial window into a different experiment. -- New fleet laws recorded (AGENTS.md + runbook): the mmproj's worst-case - VRAM is NOT inside the engine's fit projection; the first request after a - boot is a warmup artifact; a 64-token thinking budget makes a thinking - model look blind; the Bonsai-2 template reads `reasoning_effort` - (xhigh/medium/low, raises otherwise) so the harness effort fold is - load-bearing on this endpoint, and it reads `preserve_thinking`. - Recurrent DeltaNet state bills ~150 MiB **per slot** — pool arithmetic - alone misses it. -- Base alignment: **Frank chose the OFFICIAL censored payload** — publisher - provenance over refusal-freedom. The 9B it replaced was uncensored, so - the homepage card and its warning contract were re-keyed: zealot's card - now carries the base-alignment notice, the shared legal warning belongs - to dragoon alone. The CRACK re-export (byte-identical geometry per header - census) stays on disk as the one-line `MODEL_FILE` fallback if refusals - bite. -- Two live-path stragglers the cutover had to catch: - `serve-frank-zealot.sh` still DEFAULTED to the retired payload's paths - (a defaulted run would have served the old model — now re-keyed to the - shipped trio), and the OpenWebUI `model` row's `meta.description` is live - DB state (re-keyed with a DB backup; no `open-webui` restart forced — - forcing it logs every family session out for one cosmetic string). - -## 2026-10-17 — dankserver disk purge: Swift w4dn8 rollback + orphan caches (docs re-key) - -- Ask: "dump the swift rollback and the GLM GGUF — make sure the model - caches and agenttrash are empty as well." Preflight first: `lsof` empty - on both payloads, the serving trio names only the live path, and a podman - layer census proved no deleted path backs an image container (the 121 GB - layer is the ENGINE image, not a model blob). Staged, then purged with - Frank's approval: the two retired payloads, derived kvcache tiers, stale - compile caches, and every `.AGENTTRASH` on the host. -- Recovery: the retired w4dn8 rollback payload is byte-exact on the Hub — - rollback is now a **re-download, not a local file**. Corpora (grams, - kld-ref, calibration JSONL) deliberately KEPT: they are the re-convert - and KLD-gate inputs. -- Post-checks: frank-archon active and `/health` ok, serving payload - byte-size intact, the live kvcache tier untouched, no process references - a deleted path. -- Docs re-keyed: AGENTS.md fleet row + frank-archon section + both runbook - index rows; the live runbook's Rollback rewritten (HF re-fetch procedure — - the fetch script's REPO/RAD_NAME/STAGE/DEST/SIZE/SHA256 constants are - hardcoded, so retargeting at the mirror is a script edit); the retired - runbook's header corrected (was still "LIVE", never updated at - retirement). - -## 2026-10-17 — frank-dragoon cutover: native v2 abliterated payload, card's memory-saving pool - -- 2026-10-17 — frank-dragoon: cutover to the native v2 abliterated halogen - payload with the card's memory-saving KV pool; census first: the n-gram - table was ALREADY disk-backed and the only real lever was the KV pool. - See [[frank-dragoon]]. -- **Filename trap caught before the fetch:** this repo name carried two - different artifacts 32 bytes apart (the writer-transplant era's file vs - the native conversion). Byte-exact gate + LFS sha256 in - `scripts/download-frank-dragoon-v2abl.sh`, which also HARDLINKS the - peonist n-gram/vision/tokenizer trio — the release publishes only the - main file, and a hardlink is a regular file, so the no-symlink serve gate - passes without a 47.7 GiB copy. -- **Companion risk adjudicated by the engine, not by argument.** The card - does NOT establish the stock peonist companions as matched to a native - conversion and publishes no source-matched pair, so the A/B ran the - checkpoint tools on a stopped service under one knob set - (`scripts/census-frank-dragoon.sh`) plus a behaviour battery - (`scripts/battery-frank-dragoon.py`): two sound models disagreeing on - content, not a corrupt forward pass. The boot window ran through a - transient `FRANK_DRAGOON_ENV_FILE` drop-in — never a scratch port (one - iGPU, one server) — with the stop+mask law and a trap-guarded restore - (`scripts/trial-frank-dragoon.sh`). Masking fails on a relinked unit - file, as documented; stop alone was enough. -- Cost of the swap, stated plainly: two MAXIMUM-length conversations no - longer share the endpoint, the finetune's perplexity edge is gone, and - its reasoning came out compressed ~1.9×. Bought: RAM back as page cache - for the disk-backed n-gram table, faster cold prefill, clean retrieval, - and the base checkpoint itself instead of the finetune. The seam's pool - law relaxed from `pool == slots × ctx` to `ctx ≤ pool ≤ slots × ctx`. - -## 2026-10-17 — frank-dragoon: ceiling and pool cut to 200,704; crawl reset fix - -- 2026-10-17 — frank-dragoon: context ceiling and KV pool cut to 200,704 - (49 × 4096, the per-slot rung frank-zealot runs; nothing in the fleet - needed the bigger window). See [[frank-dragoon]]. -- Harness moved WITH the ceiling (400-zone law): `models.yml` - `contextWindow` = ceiling − `maxTokens`; compaction fires at 85 % of the - window, and the 180k compaction target of the 262,144 era is unreachable - because the window itself is smaller. Seam re-keyed to prove the - invariant at the new pair — `ctx ≤ pool ≤ slots × ctx` and - `harness window ≤ ctx − maxTokens` — so a future raise of one without the - other fails the gate. -- Probes on the live endpoint: over the ceiling → `400 … tokens over` in - 0.86 s — preflight rejects before any prefill, so probing costs the - endpoint nothing. -- Deploy verdict: the endpoint dispatched healthy (`/health` 200), but the - `[3/3]` Playwright gate failed on `12-crawl` and the deployer correctly - said **DEPLOY INCOMPLETE**. Root cause was NOT the endpoint — it was Plex - Web, signed out in the `qa-e2e` profile, running its own sign-in flow, - which broke the spec two different ways: (1) it redirects the top frame, - so Chromium cancelled the crawl's in-loop `page.goto("/")` with - `net::ERR_ABORTED` — and that reset sat outside the `try`, so one host - aborted the whole crawl; fixed with `backHome()`, which re-issues on that - one error class. (2) The same hand-off leaves the browser on - `app.plex.tv` (Plex's hosted surface for a session-less client), so the - `hostname == target` law could never settle; fixed with an explicit - `HANDOFFS` landing list. Plex itself was healthy throughout, and the - guard contract is still asserted for every vhost. Both symptoms recorded - as gotcha 6; the gate after both fixes went 33 passed with the crawl - green on the first attempt. - -## 2026-10-18 — issue #3 research: AMD PerfOpt retires the reason for `amd_iommu=off` - -- Ask: does a recent kernel change remove the IOMMU translation penalty - that justifies the 2026-09-11 `amd_iommu=off` boot line, i.e. can the - XDNA2 NPU come back? Answered and documented; **no host change** at the - time — the fix is not installable from the Ubuntu 26.04 archive. -- The change is **AMD PerfOpt** (spec §3.4.9, MMIO `0x016C` bit 13): merged - via the iommu tree 2026-09-24, first release **Linux 7.4**. It stops - GPA→SPA translation for the *privileged iGPU* only and leaves the IOMMU - fully armed for every other device, NPU included. Its only consumer - requires `AMD_IS_APU` **and the identity domain** — so PerfOpt rides on - top of `iommu=pt`, it does not replace it. -- **Three premises of the issue corrected.** (1) The 13–16 % is the - *vendor's* A/B against *their* previous boot line — our record copied the - figure on 2026-09-11 without ever measuring here. (2) The NPU does **not** - have to wait for 7.4: `iommu=pt` enables the IOMMU, which is all - `err=-19` asks for (`amdxdna` binds via `iommu_sva_bind_device()` into its - own SVA domain, independent of the default domain type). (3) The 460 - tok/s baseline is stale, so the window must measure `off` vs `pt` locally - and inherit no number. -- Silicon check: our IOMMU is `1022:1508` Strix/Strix Halo rev 02, and the - **Strix Halo desktop** arm measured only **2–4 %** from PerfOpt (18–23 % - on Strix Point / Krackan laptops) — the modest arm is the one that - describes this box. Kernel pick: **skip 6.18–6.18.7**, where an IOMMU - regression disables SVA outright. -- Self-correction inside the same session: the first pass claimed - `iommu=pt` alone already avoids the cost. It does not — an identity - attach keeps the page walk and enables ATS/PRI/PASID/GCR3, so `pt` - recovers **most** of the gap and PerfOpt is the last part of it. The - posted comment (Gitea issue #3) was edited to the corrected text. -- Artifacts: a new section in `dankclaw-iommu-translation-off.md` - (front-matter re-keyed to "vendor-claimed, never measured locally"), - `frank-npu.md` § *Why it is retired* corrected — the "`pt` is not a - middle ground" law is dead — the AGENTS.md key-lesson bullet re-keyed, and - a 4-step maintenance-window procedure that measures locally instead of - inheriting the vendor number. Nothing deployed — no service was touched. - -## 2026-10-18 — boot line flipped: Dankclaw now boots `iommu=pt` - -- Window executed 01:24–01:35 on Frank's go-ahead, with the A/B the - research demanded. Provenance first: `/etc/default/grub` had never been in - the repo while the IOMMU line was the most consequential kernel decision - on the box, so it is now tracked at `dankclaw/host/grub` (fetched - byte-exact) and moves only through `scripts/set-dankclaw-bootline.sh` — - diff, install 0644 root:root, `update-grub`, **verify the default - `grub.cfg` entry**, reboot on `--reboot` only. The deployer plans nothing - under `dankclaw/host/`: kernel state is not a container restart. -- **Verdict: identity DMA costs nothing on this host.** Byte-identical - traffic, +2.0 % cold prefill going OFF→PT; the 13–16 % that justified - `amd_iommu=off` for five weeks never existed here. Decode's apparent - −9.7 % was instrument noise: three repeats of the same request on the - same boot spread 31.8–47.8 t/s. -- **The XDNA2 NPU is back:** `amdxdna` bound `0000:c6:00.1`, - `/dev/accel/accel0` `crw-rw---- root:render 261,0` with `dank` in - `render`. No endpoint deployed — revival needs a quadlet with - `AddDevice=/dev/accel/accel0` + `GroupAdd=keep-groups`. -- Deliberately NOT done: the `pp_dpm_fclk` level-7 pin (it changes iGPU - timing for every workload and nothing co-runs on the NPU yet) and any - PerfOpt work (kernel `7.0.0-38` has zero `perfopt` symbols; the 26.04 - archive caps at `7.0.0-38.38`). -- Gates after boot: 0 failed units, 32 IOMMU groups (31 identity + 1 DMA = - the Genesys Logic SD reader, a kernel quirk), interrupt remapping on; - frank-dragoon's ledger byte-for-byte the same shape and Gitea / Caddy - edge / Grafana answered 200. New tool `scripts/stage-frank-dragoon-eval.sh` - rebuilds the A/B corpus byte-identically — run it through `bash -lc`, `uv` - is in `~/.local/bin`. Docs re-keyed: the decision record - `dankclaw-iommu-translation-off.md` (new § *execution*), `frank-npu.md` - (precondition satisfied, endpoint still absent), AGENTS.md key lesson. - -## 2026-10-18 — frank-zealot: Bonsai-2-9B PQ2_0 with the MTP head (3 × 262,144) -- 2026-10-18 — frank-zealot: 27B PQ2_0 → **Ternary-Bonsai-2-9B PQ2_0 with - the MTP head**, runtime moved to the PrismML vendor fork build **10777** - because the MTP arm needs the build that pairs MTP with a quantized V - cache. See [[frank-zealot]]. -- The win is the **window, not the quant**: every conversation goes back at - the trained 262,144 (the 27B era sold each slot 200,704), and the fleet's - widest context makes :7771 the **compaction hub** — `models.yml` - `contextWindow` 200704 → 262144. -- **MTP armed, then OFF by measurement.** The head is in the file and the - fork loads it, but the draft context bills its own **f16** KV over the - WHOLE pool, and at `UBATCH_SIZE=2048` that squeeze **halved decode**. The - documented MTP shape (`SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024`) was - declined — the decode gain is not worth spending the free card on it. The - arm is recorded in the env header, not left to rediscovery. -- Two bugs the trial caught, both fixed in the repo: - `${SPEC_PRESENCE_PENALTY:=0.0}` was a *default-assignment*, so the env - value always won and the documented "MTP runs without - `--presence-penalty`" arm was unreachable — now `${VAR=0.0}`, which lets - `SPEC_TYPE=draft-mtp` drop the flag. And `up{job="frank-zealot"}` had read - 0 for the **entire retained Prometheus history**: llama.cpp answers - **501** on `/metrics` unless `--metrics` is passed, so the scrape failed - beside a healthy `/health` and `LlmEndpointDown` fired for days. - `METRICS=1` is now the default and the seam pins it. -- Deployed 13:16 (`deploy.sh dankclaw --service frank-zealot`) — and the - first boot was **not the shipped payload**: a leftover - `frank-zealot.service.d/ptq1.conf` from the bench window pointed - `FRANK_ZEALOT_ENV_FILE` at `/tmp/zealot-arm-current.env`, so the unit - booted that arm. `/health` 200, `/v1/models` echoed the alias, Playwright - 33/33 — **nothing in the gate reads the payload.** Drop-in moved to - `~/.AGENTTRASH/`, `daemon-reload`, restart: the shipped trio booted. (The - drop-in was not a rogue — see the PTQ1_0 entry below.) -- Measurement law: the first battery reported 117 and 72 t/s because it - timed natural generations that stopped at 24 tokens — wall-clock aggregate - on an early EOS reads 4× low. `ignore_eos` + the engine's own `timings` - numbers is the only honest way to time this endpoint. -- OpenWebUI `model` rows re-keyed in the live DB. Note for the next edit: - the values live inside the `params`/`meta` JSON columns — the table has no - `context_length`/`description` columns — and there is no `sqlite3` binary - on the host, so `python3` stdlib + `Connection.backup()`. - -## 2026-10-18 — frank-zealot PTQ1_0 arm: measured, approved, and never written down -- **Correction to the entry above and to my own report.** The - `frank-zealot.service.d/ptq1.conf` drop-in was NOT a half-finished - experiment. The session archive (`01a14c93`) carries Frank's own - instruction at **02:36**: replace frank-zealot with - `Ternary-Bonsai-2-27B-PTQ1_0.gguf`, use the **Q8** mmproj, batch/ubatch - **3072/1536**, "fit as much context as possible in 2 contexts". -- It was the production payload **03:14:39 → 03:38:57** and served real - traffic; recovered from its own `print_timing` lines, PTQ1_0 ran - ≈47.6–49.0 t/s single-stream greedy decode (25–27 t/s per stream with two - streams live) and prefilled deep prompts faster than the PQ2_0 27B record - ever managed. -- Against the PQ2_0 27B record: PTQ1_0 freed the **1,201 MiB** its fetch - script promised and paid **−37 % decode** — the publisher's "dense - ternary unpacking loses on Ampere" warning, now confirmed on GA102. - Frank scratched the whole 27B plan at **03:33** ("scratch the current - frank-zealot plan completely… 9B… enable mtp… if you can possibly fit a - third context, do it"), which is what shipped at 13:16. -- The durable failure is mine: those numbers lived only in `/tmp` and the - journal, never in the repo, so the next agent on the host could not tell - an approved payload from a rogue drop-in and called it a hijack. The - fetch script's header now carries the measurement; - `/home/dank/models/Bonsai2-PTQ1_0/` stays on disk as the 27B rollback and - the drop-in stays trashed. - -## 2026-10-18 — frank-dragoon: the 401,408-position pool goes live; docs re-keyed to the measured boot - -- `./scripts/deploy.sh dankclaw --service frank-dragoon` — e2e 33/33, and - the drop-in check the zealot incident bought ran FIRST: `DropInPaths` - empty, so the repo env governed. 2026-10-18 — frank-dragoon: the KV pool - doubled to 401,408 positions over the 200,704 ceiling; the measured boot - ledger owns the shape. See [[frank-dragoon]]. -- Warm reuse survived the raise: one **101,987**-token prompt took 81.5 s - cold and 0.3 s on repeats — `HALOGEN_CACHE_INPLACE=1` keeps the KV rows in - the pool, so a warm turn prefills nothing and the doubling evicted nothing - the cache needed. The RAM price is still real: host `MemAvailable` fell - **26 → 22 GiB**, because every pool position is RAM the n-gram table's - page cache loses. -- Every document quoted **~10.8 GiB**, the ~28 KiB/position arithmetic; the - engine reserves **11.0 GiB** and its preflight bills **11.3**. The measured - figure wins, and every "committed but not yet deployed" clause was retired - with the measurement (AGENTS.md, the env lore block, both index pages, the - iommu page, the runbook). -- PTQ1_0 provenance corrected (see the dedicated entry): the arm that - hijacked the zealot deploy was a **repo-tracked, Frank-ordered, measured** - candidate — `scripts/download-frank-zealot-bonsai2-ptq1.sh` is committed, - and its header now carries the recovered numbers (≈48 t/s decode, 801 t/s - prefill at 141k) and the verdict: −37 % decode against PQ2_0 for 1,201 - MiB freed, then scratched by Frank's 03:33 order in favour of the 9B. - Nothing is owed on it. -- Census of the halogen HTTP surface while doing it, because the fleet bench - law quoted the wrong engine: frank-dragoon answers `/health`, `/metrics`, - `/v1/models`, `/v1/chat/completions`(POST) and 404s `/ping`, `/props`, - `/completion`, `/tokenize`; the chat response carries **no - `Server-Timing`** header. So the `/completion` `timings` object and the - `ttft;dur=` header are **llama.cpp** instruments — frank-zealot's, not - dragoon's. AGENTS.md's Benchmarking Policy re-keyed to say which engine - owns which instrument. - -## 2026-10-18 — frank-tts phase 1: the cloned-voice endpoint, named and tested (not deployed) -- Voice-over goal, phase 1 (Gitea issue #2). New service **frank-tts** on - Dankclaw: `dankclaw/voiceservice/` (stdlib-only `app.py` + `Containerfile` - + rootless quadlet), `scripts/frank-tts.env`, `tests/frank_tts_unit.bats`. - One OpenAI-shaped `POST /v1/audio/speech` surface; `voice=` is - honoured only when `/voices//reference.wav` exists, otherwise a 400 - listing the voices that are — **no silent fallback voice**; oversized - input is a 414, not a half-narrated sentence; `/health` reports backend + - voice list; no audio ever reaches the log. -- **Naming law applied (Frank's call):** treat it as a first-class model - endpoint → alias-named `frank-tts` (TTS endpoint, so not `frank-asr`), - **port 7772**, joining `7770 frank-dragoon` / `7771 frank-zealot`; the - engine upstream keeps 8880 and is deliberately not a fleet endpoint. The - internal knob names stay `SHIM_*` (they name the façade component, not the - service). -- **Voice-pack law recorded** in the env header: a pack needs - `reference.wav` (24 kHz mono, 18–25 s) **and** the pinned engine's own - speaker artifact — required, beside the WAV, never replacing it — plus - `manifest.json` provenance whose offsets are always past the first 10 s of - the source (the source-material law). Enforcement is split honestly: the - shim gates on `reference.wav` (the only engine-agnostic term); the pack - builder is what will refuse to emit a pack without its artifact. -- Landed as **PR #4** (branch `wt/20261017-181502` → `main`, merged as - `49c4e19`) — the first PR on this repo through the API, which cost three - 422s to get right: the merge endpoint takes **`{"Do":"merge"}`**, `Do` - being a *string* naming the strategy — `DoMerge`/`merge_method` do not - exist on this Gitea, and swagger JSON is disabled on the instance, so the - 422 messages are the schema. Law saved to the `gitea-api-pr-operations` - skill. - -## 2026-10-18 — TTS engine picked: arm A `qwentts.cpp`, on the live-output criterion -- Frank listened to the four audition pairs (`qwentts-*.mp3` against their - `armB-*.mp3` mainline partners) and **passed every clone**. That deleted - the only axis the A/B could not measure, and the question was reframed: - **which engine is better for live output**. On that criterion RTF only has - to stay under 1.0 with margin — the discriminators are time-to-first-audio - and run-to-run spread, because an arm whose RTF can cross 1.0 stalls a - live stream mid-sentence. -- **Chosen: arm A `qwentts.cpp` @ `5916bf7`, the resident `tts-server`, - 1.7 B talker Q4_K_M + Q8_0 tokenizer, in a `frank-tts-engine` container on - 8880.** The `TTS_BACKEND=llama-tts-cli` path in `scripts/frank-tts.env` - stays in the repo as the rollback. Three consequences are configuration, - not preference: boot-warm every voice, do not pin a cpuset (threads are - `hardware_concurrency()/2`), register every pack with its WAV (a lone - embedding is rejected). -- Two follow-ups deliberately **not** decided: the **0.6 B talker** (never - auditioned — every clip is 1.7 B) and **ICL vs embedding-only** - conditioning. Embedding-only is the shipped default because it is cheaper - and needs no `ref_text`, which this fleet cannot transcribe honestly; - `.spk` + `.rvq` still ride in every pack, so ICL remains a per-voice - opt-in for batch work where total time beats first byte. -- **Addressing law corrected** in [[Local TTS and Voice Cloning]]: the pair - may not meet at a host-loopback publish. netavark DNATs published ports - only and a bridge container cannot reach a host `127.0.0.1` bind — the - lesson that 502'd hermes — so `host.containers.internal:8880`, which - `scripts/frank-tts.env` documents, is correct only for a routable bind. - The deployed pair shares one quadlet network and the façade addresses the - engine by container name. -- **PR #12** (`feat/open-notebook-podcast` @ `114dd7a`) is **merged** as - `1eb8b21` (Frank's OK): open-notebook + SurrealDB quadlets on the private - bridge `10.89.4.0/24`, the API published on **5056** because seerr holds - `127.0.0.1:5055`, and `ESPERANTO_TTS_TIMEOUT` moved 600 → **660** to sit - one round-trip outside frank-tts's own `SHIM_TIMEOUT`. Merged is not - runnable: nothing listens on 7772 until the voice stack lands, and the - episode destination stays interim pending Gitea #7. - -## 2026-10-18 — open-notebook: the podcast generator's config lands (nothing deployed) - -- Goal: **open-notebook** as the fleet's podcast generator on Dankclaw — - notes and sources in, a multi-speaker episode out, script written by - **frank-zealot**, voices rendered by **frank-tts**, no cloud provider - anywhere in the path. Repo-side only: no host touched. Runbook: - [[Open Notebook]]. -- **Two containers, not one.** `lfnovo/open_notebook:v1-latest` plus - `surrealdb/surrealdb:v2`, both **digest-pinned** with `Pull=never` — - frank-dragoon's pinning style, because `:latest` moving underneath a - rootless service is how you get an unbootable pod after an unrelated - `systemctl-daemon-reload`. Both health probes are argument-only - (`curl -fsS …/health`, `/surreal is-ready`), and `is-ready` refuses to - pass without `SURREAL_PASS` in the environment — a credential that must - never ride the command line, so `Exec=` carries flags only. -- **Host 5055 was already spoken for** — seerr has held `127.0.0.1:5055` — - so the API publishes `0.0.0.0:5056:5055`, and because open-notebook's - frontend derives its API URL from the window origin when `API_URL` is - unset, `API_URL=http://192.168.0.143:5056` is set to match: a silent - remap would hang the UI on a port nobody listens on. 8502 publishes on - `0.0.0.0` because this service carries its own password (the - episode-audio route is unauthenticated); **8000 is published nowhere**, - where upstream exposes it on loopback with root credentials, and the seam - forbids it. -- **The CPU guard, pre-armed.** `frank-tts` spawns one ~8 GiB, 8-thread - engine per request on a box that also serves two LLMs, while - open-notebook defaults to a 25-way self-DoS; both knobs are pinned to - `1` in `scripts/open-notebook.env`. The TTS clock goes to **660 — one - round-trip outside** frank-tts's own `SHIM_TIMEOUT=600`, so a shim - give-up reads as the shim's error rather than the worker's; the seam - asserts the direction (`>`) against the frank-tts env instead of - restating a number. -- **What a credential needs in order to exist:** - `OPEN_NOTEBOOK_ENCRYPTION_KEY` must be set before the UI can save *any* - provider credential, and `OPEN_NOTEBOOK_PASSWORD` must stay **absent** to - remain auth-off — the app reads `os.environ.get(…) is not None`, so - `PASSWORD=` boots a login form asking for the empty string. The deployer - seeds `/home/dank/Containers/open-notebook/env` (600) from the host vault - and **refuses to start the pair while either secret is blank**, because a - half-configured open-notebook stores no credentials and fails quietly. -- **Episode destination is interim, by decision.** The destination is - **Gitea issue #7, `human-only`** — agents must not work it. Until Frank - settles it, the app keeps its own named volume and - `/home/dank/media/podcasts` is bind-mounted over `/app/data/podcasts` - (`DATA_FOLDER` is hard-coded `./data`), so finished MP3s are ordinary - files in his home directory instead of volume contents needing - `podman unshare`. One mount line; one deletion undoes it. -- **Two UI buttons will fail, and that is cosmetic:** Test Connection does - `GET /models` and the TTS probe synthesises voice `alloy`; - frank-tts serves `/health` and `/v1/audio/speech` only. Model rows are - created *by name* and the voice field is free text, so `narrator1/2/3` is - typed in and never discovered. Documented, not worked around. -- Three deployer seams cost time and are now law-shaped: a service must be - registered in **five** places — `SERVICE_ORDER`, `DANKCLAW_SERVICES`, the - `deploy-remote.sh` dispatch case, its pre-deploy requirement, and - `deploy.sh`'s dispatch regex, where an omission silently plans an empty - set for `--only`; a `dankclaw//*` case must precede the - `dankclaw/*.container` glob because that `*` crosses `/`; and the - changed-path routing lives *inside* the per-path loop, after the - `scripts/*.env)` case, not at the top of the file. - -## 2026-10-18 — frank-tts-engine: the arm A engine becomes deployable (nothing deployed) -- Goal: turn the engine the day's A/B picked into a **repo-owned, - boot-persistent, self-healing container** — `frank-tts-engine`, - `qwentts.cpp` commit `5916bf7c79dbb4e66fc059cdc507a79a23a28bb4`, resident - `tts-server`, container port 8880, and **not** a fleet endpoint: the - endpoint stays `frank-tts` on 7772. Runbook: [[Frank TTS Engine]]. -- **No `PublishPort` at all, which the merged A/B states as law.** netavark - DNATs published ports only, so a `127.0.0.1:8880` publish is not a path to - a peer container (the trap that 502'd hermes) — while being the one - LAN-reachable surface of an unauthenticated synthesiser. Both voice - containers therefore sit on one committed quadlet network, - `frank-tts-engine.network` at **10.89.5.0/24** (the next free /24 after - open-notebook's 10.89.4.0/24), the façade addresses the engine by - container name, and diagnostics — including the deployer's health gate - (`wait_container_healthy`, there is no URL to dial) — run through - `podman exec`. -- **The boot warmer is a unit hook, not an operator habit.** The registry is - memory-only and the first request for a registered voice decodes the whole - reference, so `ExecStartPost=-/usr/bin/podman exec frank-tts-engine - /app/warm-voices.sh` with `TimeoutStartSec=900`; the `-` prefix is what - stops a failed warm from crash-looping the endpoint under - `Restart=always`, and it logs `cold=` and `warm=` per voice on every boot. -- **Thread law recorded before anyone tunes it away:** no `--threads` - exists; `src/backend.h` uses `hardware_concurrency()/2`, so a cpuset pin - silently changes the thread count. Nothing pins this container. -- Image builds from the pinned commit with the **proven** line, verbatim — - `GGML_BLAS=OFF` forced, every GPU backend off, no `-DGGML_NATIVE` because - ggml's default is `-march=native` for the building host: the image is - built where it runs. Weights fetched by - `scripts/download-frank-tts-engine-qwentts.sh` with byte and LFS-oid - checks; the deployer gates on `--verify-only` rather than pulling 1.4 GiB - inside a deploy. - -## 2026-10-18 — frank-tts: the 7772 endpoint gets its deploy wiring, and voice naming is written down - -- Stage 2 of the voice stack: the **façade** in front of the engine is now a - deployable service too. Runbook: [[Frank TTS]]. -- **Two attached networks, in a meaningful order.** `dankclaw_default` - first — it owns the default route and is where consumers live - (open-notebook dials `http://frank-tts:7772/v1` by container name) — then - the voice pair's private bridge. `scripts/frank-tts.env`'s `TTS_UPSTREAM` - moves from `host.containers.internal:8880` to - `http://frank-tts-engine:8880`, the actual fix for a shape that could - never have worked: a bridge container cannot reach a host-loopback bind, - and the engine publishes nothing. -- **`Wants=` + `After=` on the engine, never `Requires=`:** the façade still - carries the `llama-tts-cli` rollback backend, which needs no upstream, so - an engine that will not boot must delay this endpoint, not take it down. -- The façade handler installs the **engine's** quadlet pair as well — not - tidiness: with a referenced `Network=` file absent the v1 Quadlet - generator produces no container unit at all — and it `mkdir -p - /home/dank/voices` as the deploy user, because rootless podman - auto-creates a missing bind source **root-owned** and the endpoint then - reports "no voices" instead of an error. -- **A voice is a lowercase directory name.** `^[a-z0-9][a-z0-9._-]*$`, - ≤ 64 chars, no slash/`..`/space. The shape is the intersection of two - rules that do not agree: `app.py` would accept mixed case, `tts-server` - lowercases names — so a pack named `Narrator1` resolves through the façade - and not through the engine, and the boot warmer refuses such a directory - loudly. `narrator1/2/3` are load-bearing for open-notebook. -- **No Prometheus job, on purpose.** The façade exports no `/metrics`, - Prometheus 3 refuses HTML/JSON content types (the `comfyui` and `services` - probes are documented down for exactly that), and `LlmEndpointDown` is - scoped `up{job=~"frank-.*"}` — so a `/health` scrape would page on Signal - every few minutes for a healthy endpoint. That is the frank-archon - never-green lesson pointed the other way, written in the page instead of - quietly ignored. - -## 2026-10-18 — voice-pack builder: the 10-second source-material gate becomes code - -- Third slice of the voice stack: `scripts/build-voice-pack.sh` turns a - source recording plus a crop offset into a deployable pack — - `reference.wav` (24 kHz mono `pcm_s16le`, 18–25 s), the engine's own - `reference.spk` + `reference.rvq` made from it, and `manifest.json` - provenance. Tracked tunables in `scripts/voice-packs.env`; new runbook - [[Voice Packs]]. -- **The gate is the product.** The project-wide invariant — no voice - material from the first 10 s of a sample file — had only ever been a - sentence in `scripts/frank-tts.env`. An offset inside it now exits 1 - before a directory is created, and the same assertion runs again against - the manifest that was just written, next to the recorded sha256, so a pack - cannot carry a forbidden crop even if the env file is edited into - nonsense. `10` stays a constant in the builder; the env file carries - offsets, never the law. -- **The extractor's contract was read, not guessed:** `qwen-codec` in the - pinned qwentts.cpp commit has **no `-o`** and names both artifacts next to - the input with the extension swapped. So the WAV is staged under its final - stem and `reference.wav` — the file the façade gates on — is moved into - place **last**, after the artifacts and the manifest: a failed extractor - leaves a directory that is not a voice. `--force` retires an existing pack - whole to `.AGENTTRASH/` instead of touching bytes; nothing in this script - deletes. -- Sizes are arithmetic, so they are checked: 16 codebooks × 11 bits = 22 B - per 12.5 Hz frame and `enc_dim 2048 × f32` = 8,192 B. That makes the - **effective crop window 19.84–25 s**, narrower than the WAV law's 18–25 s, - and a too-short crop is refused before the extractor runs. An engine bump - that changes either number fails the build by design. - -## 2026-10-18 — voice stack audit: three faults that no gate could see -- The voice stack had never been run end to end, so the whole path was - audited as static code against the tools it calls, and three faults came - out of it. All three are the same species: a step that **reports success - while doing nothing**. -- **The build would not have happened.** `Containerfile.engine` created its - unprivileged user with `adduser`; the pinned ubuntu:24.04 base has no - `adduser` at all — proven in the base on Dankclaw, not argued from memory. - Replacement: `useradd --system --uid 10000 --home-dir /app --shell - /usr/sbin/nologin tts`; its `SYS_UID_MAX` warning is cosmetic and is the - price of matching the façade's uid. -- **The warm pass would not have warmed.** The unit hands its env file to - podman via `--env-file`, which performs no shell parsing and takes the - literal remainder of the line, while `source` strips one quoting layer. - `WARM_TEXT` therefore reached the container with its quotes, the JSON was - invalid, and every voice would log "registered, but the warm synthesis - failed" beside an `active` unit. Fixed in the hook; the bug survived the - first draft precisely because every existing test used `source`. -- **A rebuilt image would not have been rebuilt.** `build_compose_image` - skips `podman build` on a matching `cfhash` label, and that hash covered - the `Containerfile` alone — so editing `app.py` or `tts-engine-warm.sh` - (both `COPY`'d) skipped the build while the deploy reported success and - `/health` answered with old code. Each case arm now hashes the files its - `Containerfile` copies. -- Docs carry the lessons where an operator will hit them: a **Locally built - images** section in [[deployment]], two new troubleshooting lines in - [[Frank TTS Engine]] (all-voices-cold = quoted `WARM_TEXT`; façade 502 - beside 200 = model loaded ≠ voices registered), and `Voice Packs.md` - step 4 corrected — the warm hook re-registers **every** pack on every run, - because the engine replaces a name it already holds, so a restart is what - an engine bump needs and not what a rebuilt pack needs. - -## 2026-10-18 — frank-tts-engine deployed: it builds, it serves, and the warm hook died on the kernel's argument cap -- Merged as **#14** (`747c665`) and deployed `--service frank-tts-engine`; - the image built clean — the first real proof of the `useradd` fix — and the - unit was `healthy` after 31 s. -- Then the deploy reported `0 voice(s) warm`: - `/app/warm-voices.sh: line 152: /usr/bin/curl: Argument list too long`. The - hook handed the registration body to curl as ONE argument; the kernel caps - a single argument at 131,072 bytes (`MAX_ARG_STRLEN`) and a shipped ~955 KB - `reference.wav` base64's past that, so curl never executed and the hook - logged that as "HTTP no response". Fourth silent fault of the voice stack, - same species as the other three: `/health` stayed green throughout. Fix: - the body goes to a `mktemp` file and curl gets `@file`; the regression test - is the honest shape — real listener, real `curl`, a 300 KB fixture pack — - so it fails with the same kernel error the host produced. Every shipped - pack sits well over the ~96 KB cliff, so no small fixture could ever have - seen this. -- **A trial law did not survive deployment.** Against an empty registry the - first synthesis of a freshly registered voice cost ~1.9 s, the next ~1.2 s. - The trial's 6.2–8.0 s cold / 0.5 ms restore ladder, which the warmer's own - header repeated, is retired: warming buys registration — mandatory, because - the registry is memory-only — plus about **0.6 s**. Corrected in - [[Frank TTS Engine]] law 1 and its troubleshooting table, and in - [[Local TTS and Voice Cloning]] and [[Voice Packs]]. -- Sweep afterwards: the retired ladder still lived in six places the first - pass missed — both quadlet files, `scripts/frank-tts-engine.env`, - `scripts/frank-tts.env`, the [[Frank TTS]] troubleshooting row, the - research brief's consequences paragraph, and one test comment. All - re-keyed to the measured claim. The trial entry earlier in this log is - left as written — a dated worklog entry is a record — and this line is the - correction that points at it. - -## 2026-10-18 — 7772 is live and the harness speaks through it -- Merged **#15** (`0315a62`, the warmer's argument-cap fix) and **#16** - (`d2ce669`, the comment sweep), then deployed both voice services in order - (`--service frank-tts-engine`, then `--service frank-tts`); Playwright gate - 33/33 twice. Deployed-state proof from a LAN client: `/health` → - `omni-http` with all three voices, and the same sentence in `narrator1` vs - `narrator2` came back with different sha256 and different RMS — the name - actually selects a voice. -- **The harness `speech` role moved off Kokoro**: `config.yml` - `modelRoles.speech: frank-tts (dankclaw)/frank-tts`. No voice key moves - with it. -- Two harness laws learned by breaking it first (both now on [[Frank TTS]]): - an explicit speech role has **no fallback candidate**, so a dead endpoint - errors instead of degrading; and `models.yml` loads at process start while - `config.yml` is read live, so a session older than the `frank-tts` row - fails exactly like a dead endpoint. -- Corrected the same evening, by breaking it in a restarted session: a - harness call that names no voice reaches 7772 with **no `voice` key at - all** — `400 unknown voice None; available voices: …` — because no setting - feeds one. The endpoint therefore grew `DEFAULT_VOICE` (its own PR): a - voice-less request speaks the declared pack, an unknown name still 400s - with the list. `speech.voice: narrator1` — briefly recorded as wiring — - went back to `am_fenrir`: that key is the vocalizer's Kokoro enum, so a - pack name there breaks speak-aloud and does nothing for synthesis. -- Status sweep: six pages still said the endpoint did not exist — both vault - descriptions, the two `index.md` bullets, the Open Notebook blocker (the - voice leg works; open-notebook itself is what is missing), the engine - page's deploy order (now a deploy record), and the homepage card, which - lost `wip: down` so the admin panel probes 7772 for real. - -## 2026-10-18 — the default voice shipped: a harness voiceover needs no voice -- Merged **#17** (`b590695`) and **#18** (`7889a68`) and deployed - `--service frank-tts --service homepage` in one dispatch, Playwright gate - 33/33. The call that had failed is the one that proves the fix: same - device, **no** `voice_id` — `Saved 430124 bytes (…, voice=default, - codec=wav)` — and the façade journal says what actually spoke, - `speech voice=narrator1 (default)`. The harness prints `voice=default` - because it never had a name to print; the voice that answered is the - operator's `DEFAULT_VOICE` line in `scripts/frank-tts.env`, and the journal - keeps them distinguishable forever after. -- The page gained a **Default voice** row in its contract table and - [[#Naming a voice]] now states that a call naming nothing is the harness's - normal shape, not a client bug — the misconception that produced the 400 in - the first place. - -## 2026-10-18 — OpenWebUI "voice narrator1 not found": the rejector is in the browser -- Frank's toast — `Error: Voice "narrator1" not found. Should be one of: - af_heart, …, bm_fable` — looked like frank-tts refusing the name. It is - not: that 29-name enum is the voice pack of **Kokoro.js running in the - page** — OpenWebUI 0.11.4 ships that implementation in its own front end. -- Instruments that separated the two paths, in order: the admin config DB is - correct (`audio.tts.engine=openai`, base `http://host.containers.internal:7772/v1`, - `voice=narrator1`); from inside the container the exact OpenWebUI body - returns **200 `audio/mpeg`**; and `podman logs open-webui` holds **zero** - `audio/speech` requests for the failing click. A server-path failure leaves - a log line and a frank-tts error — a `browser-kokoro` failure leaves - neither — and no container env or `user.settings` row carries an override, - so the engine choice is per-browser storage, not fleet state. -- Fix is a UI click, not a deploy: Settings → Audio → TTS Engine → - **Default** hands synthesis back to frank-tts and the cloned voices; - choosing `af_heart` keeps in-browser Kokoro, which cannot clone (fixed - pack). Recorded on [[OpenWebUI]] under *Text-to-speech (frank-tts)*. -- Also in the window: **PR #22 merged** (`0c7fe8b`, `dank/command-center`) so - `agent/extensions/dankvoice.ts` landed on the auto-load path — proven in a - plain `omp` session with no flags: `/say merged build check` → WAV to the - player, temp dir cleaned. Sessions open before the merge need a restart to - pick the command up. - -## 2026-10-18 — frank-zealot: 2 × 262,144 over unquantized f16 KV -- 2026-10-18 — frank-zealot: shape cut to 2 × 262,144 over unquantized f16 KV - (the 786,432 pool cannot fit exact KV on the 3090; concurrency 3 → 2, and - MTP is now doubly out — off by the 13:16 measurement, and exact KV bills a - draft-context f16 pool over the whole context). See [[frank-zealot]]. -- Arithmetic correction to the repo's long-standing figure: "the f16 pair - would need **26,112 MiB**" doubled the q8_0 number *including* its 1/32 - scale overhead — q8_0 is 17,408 B/token (1.0625 B/element), so f16/q8_0 is - **1.882×**, not 2×, and exact f16 over 786,432 is **24,576 MiB**. The - conclusion stands (three f16 windows cannot boot this card); the number was - wrong. -- New gate in `tests/frank_zealot_unit.bats`: dtype × pool plus the measured - non-KV terms must fit the driver-addressable card with **≥ 1 GiB** of slack - — and the same gate proves it refuses the impossible arm (f16 at 786,432), - not merely passes the shipped one. From 483fd843b74ec018f6300ff8bec2fb7d0646b37a Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 19 Oct 2026 05:08:22 -0700 Subject: [PATCH 04/12] fix(scripts): port-census must not call prose words facts A hyphen or a trailing period inside a word was enough to make it a tracked token, so `long-standing`, `deployed:` and `cleaned.` were reported as lost facts while real identifiers hid behind them. Require a digit, dot, slash, colon, `--`, two hyphens (`frank-tts-engine`) or ALL-CAPS in the stripped word. The census of the log rewrite went from 979 noisy lines to a set whose unheld tokens are all fact-shaped. --- scripts/port-census.py | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/scripts/port-census.py b/scripts/port-census.py index 8ea1f66..3c38846 100755 --- a/scripts/port-census.py +++ b/scripts/port-census.py @@ -25,8 +25,14 @@ def toks(line): out.append(n) # Identifiers, flags, paths and backticked spans are the facts worth # tracking; ordinary prose words are not — they reword on every rewrite. - out += [w for w in re.findall(r"[A-Za-z][\w./:-]{7,}", line) - if re.search(r"[\d_.:/-]", w) or w.isupper()] + # A hyphen alone is not evidence of an identifier (`long-standing`), so a + # dashed word counts only with two hyphens (`frank-tts-engine`) or with a + # digit, dot, slash, colon or `--` inside it. + for w in re.findall(r"[A-Za-z][\w./:-]{7,}", line): + core = w.strip(".,;:!?\"')]") + if (core.startswith("--") or core.isupper() + or re.search(r"[\d_/:.]", core) or core.count("-") >= 2): + out.append(core) out += [s.strip() for s in re.findall(r"`([^`]+)`", line) if len(s) > 3] return out From 293c25c50bb139fb67a9597e13bb9026f781d140 Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 19 Oct 2026 05:08:29 -0700 Subject: [PATCH 05/12] docs(agents): compress the load-bearing lessons, repoint their owner links MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four bullets in Key Lessons Learned carried narrative a vault page owns: the 2026-09-20 board swap ([[Dual-R9700-Cooperation]]), the `--override-tensor` gate ([[Llama Cpp Multi-Device Splits]]), the x4 link budget ([[x4-pair-experiment]]), and the silent-RAM incident plus the Prometheus `--metrics` law ([[frank-archon]], [[frank-zealot]]). Each keeps its law, its numbers and its error strings; the causal narration that duplicated a page is what went. The RAM bullet linked to [[log]] — the incident is owned by [[frank-archon]], so the link now names the owner. --- AGENTS.md | 69 ++++++++++++++++++++++++++----------------------------- 1 file changed, 32 insertions(+), 37 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index b41f980..6fa63a9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -485,13 +485,12 @@ linked runbooks. 6 set. AMD desktop platforms put one card on CPU root ports (atomics work) and the rest behind the chipset (usually not) — that asymmetry is what blocked TP=2 here. A one-card collective proves nothing: NCCL short-circuits at `world_size=1`. [[PCIe Link and Atomics]] -- **That wall was physical, and a board swap removed it on 2026-09-20.** dankserver took the path of - a board whose CPU lanes bifurcate to x8/x8 Gen5, both cards on root ports, every bridge `Routing+` - and BOTH endpoints `ReqEn+` — the exact precondition the old test plan named. Bandwidth was never - the constraint (a 27B TP=2 allreduce at batch 4 is ≈10 MB per token-step, under 0.4 ms at Gen5 x8). - The collective choice matters: TheRock's RCCL builds lack gfx1201 device code - (`invalid device function`), which is why vLLM runs its R4D one-shot allreduce here at all. - Evidence matrix and the measured three-arm verdict: [[Dual-R9700-Cooperation]]. +- **That wall was physical, and a board swap removed it on 2026-09-20.** dankserver took a board + whose CPU lanes bifurcate to x8/x8 Gen5: both cards on root ports, every bridge `Routing+`, BOTH + endpoints `ReqEn+`. Bandwidth was never the constraint (a 27B TP=2 allreduce at batch 4 is ≈10 MB + per token-step, under 0.4 ms at Gen5 x8); the collective choice is — TheRock's RCCL builds lack + gfx1201 device code (`invalid device function`), which is why vLLM runs its R4D one-shot allreduce + here at all. Evidence matrix and the three-arm verdict: [[Dual-R9700-Cooperation]]. - **When a GPU collective dies, read the runtime, not the framework.** The message that identified the cause came from `AMD_LOG_LEVEL=3` (`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch failed!`), never from vLLM or RCCL — the env var costs one line and prints the @@ -499,26 +498,23 @@ linked runbooks. - **Layer split is the no-collective path, on one vendor's cards and across two — and `--override-tensor` is what cannot ride with it.** Layers go in order and only the boundary activation crosses PCIe once per forward pass, so it works wherever a collective cannot be - dispatched (and on an RCCL-less fork it beats tensor mode, which falls back to a butterfly - allreduce); it is also the cross-vendor answer, where a CUDA + Vulkan split held a real - two-device graph. But `--override-tensor =` sets `has_tensor_overrides()`, a - REQUIRED term of the scheduler's `pipeline_parallel` flag (llama-context.cpp), and that flag sets - `n_copies` in `ggml_backend_sched_new`: with `n_copies == 1` a pre-allocated weight on a - non-primary device has no backend and the graph aborts (ggml-backend.cpp, reported with the leaf's - own op name `(NONE)`). `=CPU` is the only override target that works, because the CPU backend is - last in the list, and pure Vulkan changes nothing — the gate is the override, not the driver mix. - Pin the spec-decode drafter to one device: an unpinned draft under tensor mode aborts the KV build, - and on a split its home is the card holding the target's last layers. - [[Llama Cpp Multi-Device Splits]] [[x4-pair-experiment]] + dispatched (on an RCCL-less fork it beats tensor mode, which falls back to a butterfly allreduce), + and it is the cross-vendor answer too (a CUDA + Vulkan split held a real two-device graph). But + `--override-tensor =` sets `has_tensor_overrides()`, a REQUIRED term of the + scheduler's `pipeline_parallel` flag, and that flag sets `n_copies`: with `n_copies == 1` a + pre-allocated weight on a non-primary device has no backend and the graph aborts (reported with + the leaf's op name `(NONE)`). `=CPU` is the only override target that works — the CPU backend is + last in the list — and pure Vulkan changes nothing: the gate is the override, not the driver mix. + Pin the spec-decode drafter to one device; on a split its home is the card holding the target's + last layers. [[Llama Cpp Multi-Device Splits]] [[x4-pair-experiment]] - **A host↔3090 link at x4 measures 6.72 GB/s, and that is a design constraint, not a detail.** - Pinned H2D 6.72 / D2H 6.60 GB/s (≈85 % of the Gen4 x4 ceiling), async no better than sync and - pageable within 1 % of pinned — the copies are LINK-limited, not staging-limited, so the usual - "pageable costs 2×" rule does not apply. The card and the port beneath it are both wider than x4, - so the x4 lives in the adapter or cable, and the link idles at 2.5 GT/s until load. Consequence - for any engine that STREAMS weights across it: a 3 MiB slab costs 474 µs, so a 111 GiB A3B model - needing ~1.70 GiB of active experts per token lands at 3.7 tok/s, against 20.46 tok/s for the same - model with the experts COMPUTED in host RAM (`-ot exps=CPU`, where the link carries activations - only). Streaming over PCIe loses; computing in RAM does not care. [[x4-pair-experiment]] + Pinned H2D 6.72 / D2H 6.60 GB/s (≈85 % of the Gen4 x4 ceiling), pageable within 1 % of pinned — + the copies are LINK-limited, so the usual "pageable costs 2×" rule does not apply; the x4 lives in + the adapter or cable (card and port are both wider). For any engine that STREAMS weights across + it: a 3 MiB slab costs 474 µs, so a 111 GiB A3B model needing ~1.70 GiB of active experts per + token lands at 3.7 tok/s against 20.46 tok/s with the experts COMPUTED in host RAM + (`-ot exps=CPU`, where the link carries activations only). Streaming loses; RAM does not care. + [[x4-pair-experiment]] **ROCm / GPU** @@ -588,12 +584,12 @@ linked runbooks. can boot before a human reaches it. [[frank-archon]] - **Silent RAM corruption outranks every software theory.** frank-archon's long-conversation repetition/garbage was marginal DDR5 at 4800 MT/s on four DIMMs, fixed by dropping to 4600. - Consumer non-ECC memory raises NO MCE for a silent flip, and a short in-band pattern check - (60 s over 8 GiB of 246 GiB) is not a stability proof — combined, those two instruments said - "ruled out" while the RAM was the cause, and "4800 is the JEDEC floor" is a rating, not a margin - guarantee. When the fault is data-shaped (NaN states, garbage tokens) and a software revert does - not move it, price in the RAM before the runtime — and remember a revert tested under the same - marginal hardware is itself confounded. [[log]] + Consumer non-ECC raises NO MCE for a silent flip, and a short in-band pattern check (60 s over + 8 GiB of 246 GiB) is not a stability proof — both instruments said "ruled out" while the RAM was + the cause, and "4800 is the JEDEC floor" is a rating, not a margin guarantee. When the fault is + data-shaped (NaN states, garbage tokens) and a software revert does not move it, price in the RAM + before the runtime; a revert tested under the same marginal hardware is itself confounded. + [[frank-archon]] **Runtime-specific guards** @@ -724,11 +720,10 @@ linked runbooks. `CMakeCache.txt` — never from the build flags you remember passing. [[frank-zealot]] - **A llama.cpp endpoint is invisible to Prometheus unless `--metrics` is passed**: without it the engine answers **501 Not Implemented** on `/metrics`, so the scrape fails while `/health` - answers 200 and `LlmEndpointDown` fires beside a healthy service — that is how `frank-zealot` - sat red across four payload generations. halogen serves `/metrics` natively, so the asymmetry - reads as "only the 3090 is broken". Diagnose with `lastError` from `/api/v1/targets`, not the - alert list, and note Alertmanager has no persistence: a reboot re-stamps `startsAt` on an alert - that has fired for weeks. [[Observability]] + answers 200 and `LlmEndpointDown` fires beside a healthy service. halogen serves `/metrics` + natively, so the asymmetry reads as "only the 3090 is broken". Diagnose with `lastError` from + `/api/v1/targets`, not the alert list, and note Alertmanager has no persistence: a reboot + re-stamps `startsAt` on an alert that has fired for weeks. [[Observability]] [[frank-zealot]] - **A leftover bench drop-in silently hijacks the next deploy, and no health gate sees it.** The sanctioned bench-window method (a transient `Environment=FRANK__ENV_FILE=/tmp/.env` drop-in sourcing the repo env) outlives the bench unless retired: a leftover From fb8862b24cef9994687c04aa4319ddedebb2da42 Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 19 Oct 2026 05:08:47 -0700 Subject: [PATCH 06/12] docs(vault): verified corrections with sources, not assertions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - frank-archon KV dtype law: `bf16|fp8` is a **radiance** restriction. Stock vLLM advertises `auto|fp8|fp8_e5m2|fp8_e4m3` (docs.vllm.ai engine args), and even there the advertised set is not the usable set — several accepted values crash engine init at the attention layer's own dtype check (vllm-project/vllm#42587). The page stated the restriction as if upstream had it. - Open Notebook: re-censused 2026-10-19 — no unit file, no installed quadlet, no container. The page already said NOT deployed; it now says what was checked, so the claim is re-checkable instead of remembered. --- serverAppConfigs-docs/Dankclaw/Open Notebook.md | 2 +- serverAppConfigs-docs/LLM/frank-archon.md | 6 +++++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/serverAppConfigs-docs/Dankclaw/Open Notebook.md b/serverAppConfigs-docs/Dankclaw/Open Notebook.md index 04f4cd1..c9911f1 100644 --- a/serverAppConfigs-docs/Dankclaw/Open Notebook.md +++ b/serverAppConfigs-docs/Dankclaw/Open Notebook.md @@ -13,7 +13,7 @@ timestamp: 2026-10-18 open-notebook ([lfnovo/open-notebook](https://github.com/lfnovo/open-notebook)) is the notebook layer that turns existing notes and sources into a **spoken episode**: an outline, then a dialogue transcript, then one synthesized clip per speaker turn, concatenated with ffmpeg into an MP3. In the fleet it is the consumer that closes the loop Frank's audio stack was built for — the script is written by **frank-zealot**, the voices are rendered by **frank-tts**, and nothing leaves the LAN. -**It is NOT deployed.** This page documents repo-owned configuration — the Quadlet units, the env file, the deployer path, the homepage card — and the contract those files were written against. On 2026-10-18 no `open-notebook` or `open-notebook-surrealdb` container exists on Dankclaw (`podman ps -a` census, 2026-10-18), and the homepage card ships `wip: "down"` with `showLink: false`, so the card is hidden and reported `wip-down`. Deploying it is a deliberate next step, not a pending one. +**It is NOT deployed.** This page documents repo-owned configuration — the Quadlet units, the env file, the deployer path, the homepage card — and the contract those files were written against. On 2026-10-18 no `open-notebook` or `open-notebook-surrealdb` container exists on Dankclaw (`podman ps -a` census, 2026-10-18), and the homepage card ships `wip: "down"` with `showLink: false`, so the card is hidden and reported `wip-down`. Re-censused 2026-10-19 and unchanged: no `*notebook*` unit in `systemctl --user list-unit-files`, nothing installed under `~/.config/containers/systemd/` by that name, no container in `podman ps` — the repo's three quadlet files have never been installed on the host. Deploying it is a deliberate next step, not a pending one. **Host:** Dankclaw (rootless Podman, user `dank`) — **Ports:** 8502 UI and 5056 API, published on the LAN, **no vhost and no Caddy route** — **Deployer:** `open-notebook` (one service, two units) — **Docs:** https://github.com/lfnovo/open-notebook diff --git a/serverAppConfigs-docs/LLM/frank-archon.md b/serverAppConfigs-docs/LLM/frank-archon.md index d5856f2..7741348 100644 --- a/serverAppConfigs-docs/LLM/frank-archon.md +++ b/serverAppConfigs-docs/LLM/frank-archon.md @@ -150,7 +150,11 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10). ### The radiance engine - **KV dtype law.** `--kv-cache-dtype` takes only `bf16|fp8` — ONE dtype for the whole cache, - so llama.cpp's exact-K / quantized-V pair (what [[frank-zealot]] runs as `--cache-type-k`/ + a **radiance** restriction, not stock vLLM's: upstream advertises `auto|fp8|fp8_e5m2|fp8_e4m3` + (`docs.vllm.ai` engine-args) and even there the advertised set is not the usable set — several + accepted values crash engine init at the attention layer's own dtype check + ([vllm-project/vllm#42587](https://github.com/vllm-project/vllm/issues/42587)). + So llama.cpp's exact-K / quantized-V pair (what [[frank-zealot]] runs as `--cache-type-k`/ `--cache-type-v`) is unservable here: the rung below bf16 is fp8 on both halves. Measured per-rank cost out of the boot ledger's group table (fp8 boot 2026-10-08, bf16 boot 2026-10-11): `kv_attn` 13 layers / 4-token blocks pages **26.00 → 52.00 KiB**, while From c860fb3c8c70e85950483128cc55d9160ea7e30a Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 19 Oct 2026 05:09:48 -0700 Subject: [PATCH 07/12] docs(vault): the frank-medium locator's two wrong cells MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The ROCmFPX row said "single card, pinned at 36e9acd": `36e9acd` is the SHA that `scripts/build-rocmfpx.sh` pins for the *build*, and the era page records the deployed commit as `c49ebdb` running a 2× R9700 `-sm layer` split over `ROCm0,ROCm1` at 131,072 × 1 slot with draft-MTP pinned to `ROCm1`. The 2026-09-20 row skipped a payload generation: the same day the endpoint swapped MXFP4 → FP8 E4M3 128×128 ([[frank-medium-model-swap]]) before MXFP6 arrived on 09-21. Both cells now quote the era pages. --- serverAppConfigs-docs/LLM/frank-medium.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/serverAppConfigs-docs/LLM/frank-medium.md b/serverAppConfigs-docs/LLM/frank-medium.md index 98a906e..807aaf8 100644 --- a/serverAppConfigs-docs/LLM/frank-medium.md +++ b/serverAppConfigs-docs/LLM/frank-medium.md @@ -26,10 +26,10 @@ Windows are quoted from each era page's own banner; no date here is derived. | Window | Runtime | Payload | Shape | Page | Rollback | |---|---|---|---|---|---| -| 2026-09-15 → 2026-09-16 | llama.cpp, ROCmFPX fork pinned at commit `36e9acd` | `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` (custom tensor type 103) | single card; the fork is the only loader | [[frank-medium-rocmfpx]] | re-fetch the GGUF, rebuild the fork | +| 2026-09-15 → 2026-09-16 | `charlie12345/ROCmFPX` llama.cpp `main` at `c49ebdb`, self-contained ROCm 7.14 | `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` (custom tensor type 103) + an MTP Q4_0 draft | 2× R9700 `-sm layer` over `ROCm0,ROCm1`, 131,072 × 1 slot, f16 KV + flash attention, draft-MTP pinned to `ROCm1`; the fork is the only loader for that tensor type | [[frank-medium-rocmfpx]] | re-fetch the GGUF, rebuild the fork | | 2026-09-16 → 2026-09-19 | vllm-radiance on the old TUF GAMING B650-E board | ParoQuant MXFP4 | two single-GPU replicas behind an adaptive-routing proxy | [[frank-medium-radiance-mxfp4]] (sections below its deployed-shape header) | the replica pair is reconstructable from that page's flags | | 2026-09-16 → 2026-09-20 | llama.cpp `fb27a525`, ROCm `7.14.0a` | its own GGUF build, DFlash drafter | `-sm layer` split across the two cards, drafter pinned to one device | [[frank-medium-llamacpp-dflash2]] | the page is the named rollback target for the llama shape | -| 2026-09-20 → disputed, see Open questions | vllm-radiance on the replacement ProArt board | ParoQuant MXFP4, then MXFP6 (`paroquant_mxfp6`) with bf16 KV | `--tensor-parallel-size 2` over x8/x8 Gen5 root ports | [[frank-medium-radiance-mxfp4]] (deployed-shape header) | re-fetch the MXFP4 checkpoint (host census 2026-10-19: not local); image `localhost/radiance-vllm:0.9.3` | +| 2026-09-20 → disputed, see Open questions | vllm-radiance on the replacement ProArt board | ParoQuant MXFP4 → FP8 E4M3 128×128 (`orcarouter/…-FP8`, same day, [[frank-medium-model-swap]]) → ParoQuant MXFP6 (`paroquant_mxfp6`) from 09-21 | `--tensor-parallel-size 2` over x8/x8 Gen5 root ports | [[frank-medium-radiance-mxfp4]] (deployed-shape header) | re-fetch the MXFP4 checkpoint (host census 2026-10-19: not local); image `localhost/radiance-vllm:0.9.3` | `[[frank-medium-batch-ubatch]]` is not an era: it is the batch-versus-ubatch sweep run on the layer-split pair at its deployed shape, and its numbers belong to that From 2a124876538f9cc612378a7ad4c046b0f521dad0 Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 19 Oct 2026 09:58:04 -0700 Subject: [PATCH 08/12] fix(vault): delete the restated consolidation page, not the vault `Multi-Node Runtimes.md` (+257) restated launch material that [[Qwen3.8 PP Deployment]] (build target, two-process RPC table, cache-prune cron), [[Sample Script Set]] (runnable scripts), [[Ethernet Bandwidth Math]] and [[Expected Performance]] already own, and mixed-vendor numbers owned by [[x4-pair-experiment]]. Consolidation means the facts end up in ONE place, not that a new page is written beside the old owners: the four stubs it replaced were frontmatter-only (9 lines each) and needed no successor. Page trashed, three inbound links repointed at the real owners. --- .../Expected Performance.md | 4 +- .../Multi-Node Runtimes.md | 257 ------------------ .../Pipeline Parallelism/Sample Script Set.md | 2 +- 3 files changed, 3 insertions(+), 260 deletions(-) delete mode 100644 serverAppConfigs-docs/LLM/Pipeline Parallelism/Multi-Node Runtimes.md diff --git a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Expected Performance.md b/serverAppConfigs-docs/LLM/Pipeline Parallelism/Expected Performance.md index 4fc57e2..201b5ca 100644 --- a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Expected Performance.md +++ b/serverAppConfigs-docs/LLM/Pipeline Parallelism/Expected Performance.md @@ -13,7 +13,7 @@ timestamp: 2026-08-14 This page estimates the speed of a 25-40B model split across two machines over ethernet. It uses the bandwidth math in [[Ethernet Bandwidth Math]] -and the mixed-vendor measurements in [[Multi-Node Runtimes]]. +and the mixed-vendor measurements in [[x4-pair-experiment]]. ## The Bandwidth Ceiling Is Not the Limit @@ -93,5 +93,5 @@ roughly 200k-260k tokens across both nodes. ## See Also - [[Ethernet Bandwidth Math]] -- [[Multi-Node Runtimes]] +- [[Qwen3.8 PP Deployment]] - [[Sample Script Set]] diff --git a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Multi-Node Runtimes.md b/serverAppConfigs-docs/LLM/Pipeline Parallelism/Multi-Node Runtimes.md deleted file mode 100644 index b99a4e6..0000000 --- a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Multi-Node Runtimes.md +++ /dev/null @@ -1,257 +0,0 @@ ---- -type: reference -title: Multi-Node Runtimes -description: "How each two-machine runtime is built and launched, what it can combine with, and the mixed-vendor verdicts." -tags: - - pipeline-parallelism - - llamacpp - - vllm - - speculative -timestamp: 2026-10-19 ---- - -# Multi-Node Runtimes - -Two runtimes can put one model on two machines: llama.cpp over the ggml-rpc -backend, and vLLM with pipeline parallelism. This page owns the cross-runtime -facts — how each is built and launched, what each can and cannot be combined -with, and the mixed-vendor verdicts. Facts owned by a sibling page are linked, -not restated. Bytes-per-token arithmetic and link budgets belong to -[[Ethernet Bandwidth Math]], throughput estimates to [[Expected Performance]], -the real deployment and its verified gotchas to [[Qwen3.8 PP Deployment]], the -educational scripts to [[Sample Script Set]], drafter mechanisms to -[[DFlash DSpark Speculative Decoding]], single-host multi-GPU placement to -[[Llama Cpp Multi-Device Splits]]. - -## What each runtime is - -Both distribute by **layers**, not inside layers. - -| | llama.cpp over ggml-rpc | vLLM pipeline parallelism | -| --- | --- | --- | -| unit of distribution | a contiguous slice of layers (`--split-mode layer`, the default) | a contiguous slice of layers (`--pipeline-parallel-size`) | -| process model | one `llama-server`/`llama-cli` on the head; a `ggml-rpc-server` per worker | one `vllm serve` per node, one engine across them | -| what crosses the wire | whatever graph work the head offloads to the remote devices | the hidden state at each stage boundary, point to point | -| KV cache | distributed with the weights; the KV of a layer lives with the owner of that layer | each stage holds the KV of the layers it owns | -| speculation together with the split | yes in llama.cpp (see below) | no, per vLLM's own docs | - -Upstream's own summary is why layer-wise is the ethernet choice: pipeline -parallel "minimizes data transfers between GPUs but requires many tokens to -scale well", while tensor parallel "does multiple cross-GPU reductions per -layer … much more bottlenecked by the GPU interconnect speed". - -Do not carry the tensor path across machines: `--split-mode tensor` is -experimental, needs flash attention, rejects quantized KV, and is not -implemented for the MoE and hybrid architectures the guide lists — and -per-tensor placement across vendors is not expressible (§1 of [[Llama Cpp Multi-Device Splits]]). - -## llama.cpp over ggml-rpc - -### Build and launch - -Both sides take `-DGGML_RPC=ON` next to that host's own backend: - -```bash -cmake .. -DGGML_CUDA=ON -DGGML_RPC=ON # a CUDA worker; -DGGML_HIP=ON for an AMD one -cmake --build . --config Release -``` - -Worker side — `ggml-rpc-server` exposes ggml devices on the remote host: - -```bash -bin/ggml-rpc-server # endpoint 127.0.0.1:50052 by default -bin/ggml-rpc-server --device CUDA0 -c # or CUDA_VISIBLE_DEVICES=0; -c = local cache -``` - -- It exposes every accelerator it can see; a host with no accelerator exposes a - single `CPU` device. `--device` and `CUDA_VISIBLE_DEVICES` are equivalent - ways to narrow that set. -- `-c` keeps large tensors in `$HOME/.cache/llama.cpp/rpc` (redirect with - `LLAMA_CACHE`) so a big model does not re-stream per load; that cache - filling the worker disk is a recorded fleet incident ([[Qwen3.8 PP Deployment]]). - -Head side — build the local backends plus `-DGGML_RPC=ON`, then name the -workers: - -```bash -llama-server -m model.gguf --rpc 192.168.88.10:50052,192.168.88.11:50052 -ngl 99 -``` - -`--rpc` is a comma-separated list of `host:port`, one entry per -`ggml-rpc-server`; the head's own devices stay in the pool next to the remote -ones. Launch scripts for this shape: [[Sample Script Set]]. - -### Split, context, KV - -- Default placement: weights **and** KV across all devices, local and remote, - "in proportion to each device's available memory". Override with - `--tensor-split`, whose proportions follow the order in `--device`. Why the - fleet always pins it (a Strix reports a huge GTT window, so the automatic - split lands on the slow device) is [[Qwen3.8 PP Deployment]]'s. -- `-sm layer` (the default) is pipeline parallelism: each device holds a - contiguous slice of layers and the KV cache of a layer lives on the device - that owns that layer. -- KV types are the usual `-ctk` / `-ctv`, `f16` by default. `--fit` is on by - default and sizes unset arguments to device memory; turn it off to hold a - context you set on purpose. Context is one global `--ctx-size` — the RPC - README documents no per-node context or per-node KV knob. -- Upstream's out-of-memory ladder (stated for tensor mode, useful order - anyway): lower `--ctx-size`, then `--parallel`, then `--n-gpu-layers`. -- Transport is TCP; RDMA is negotiated at the handshake with no flag changes - and falls back to TCP unless both peers can use it (Linux RoCEv2 via - `libibverbs`, macOS RDMA over Thunderbolt 5 via `librdma`). It is point to - point — name the peer's RDMA-capable address in `--rpc` or the session stays - on TCP. `GGML_RPC_NO_RDMA=1` forces TCP; `GGML_RPC_DEBUG` traces the backend. - -Upstream calls the backend a proof of concept, "fragile and insecure" (see the -Security section). Two limits shape the design: decode is bound by the per-step -round trip, not by the link's bytes ([[Ethernet Bandwidth Math]]), and every -node must run the identical build or the handshake hangs ([[Qwen3.8 PP Deployment]]). - -## vLLM pipeline parallelism - -> **Provenance.** Flags and mechanics below were recorded 2026-08-14 and are -> not re-checked upstream in this pass (see "Unverified in this pass"); the -> incompatibility lines, the method catalog and the issue history are re-verified. - -- `--pipeline-parallel-size` divides the layers into contiguous stages; - `--tensor-parallel-size` still works inside a node, so the canonical two-node - shape is tp = GPUs per node, pp = nodes. -- Two backends: Ray (`--distributed-executor-backend ray`, the worker joining - a `ray start --head`) or multiprocessing (`--distributed-executor-backend mp` - with `--nnodes`, `--node-rank`, `--master-addr`, `--headless` on the - follower). `mp` avoids the Ray dependency; Ray brings the cluster manager. -- What the interconnect must carry: one activation tensor per stage boundary - per forward pass, point to point — not a per-layer reduction. That is the - whole cross-node traffic story; the byte arithmetic and link budgets are - [[Ethernet Bandwidth Math]]. Pin the NIC vLLM talks on - (`NCCL_SOCKET_IFNAME`) rather than letting it guess an interface. -- The PP bubble: with N stages and one sequence in flight, one stage works - while N-1 idle, so pp=2 costs half the machine at batch one. Batching is the - cure — pipeline parallelism "requires many tokens to scale well" — and the - microbatch formula is [[Ethernet Bandwidth Math]] §5.3. -- KV and context: each stage keeps the KV of its own layers, so the pool grows - with the stage count, and `--max-model-len` is not reduced by the split - (recorded 2026-08-14, unverified here). -- What it cannot combine with, in vLLM's words: "Pipeline parallelism is not - composable with speculative decoding as of `vllm<=0.15.0`", and "Speculative - decoding with a draft models is not supported in `vllm<=0.10.0`". Composing - the two has been requested since 2024 and closed stale (#6911, #10615); PP - also carries a Qwen3.5 bug report closed stale without a fix (#36643). -- Method catalog at v0.25.1: EAGLE, MTP, draft model, PARD, MLP, n-gram, suffix - decoding, hidden-state extraction, a custom proposer class, dynamic - speculative decoding; `dflash` is an accepted `method` value. - -## Speculative decoding across the split - -llama.cpp chains several implementations behind one comma-separated -`--spec-type` list; the split does not change the chain, only where its passes -run. Mechanisms, chaining laws and drafter KV sizing are -[[DFlash DSpark Speculative Decoding]]'s; only the split interaction is here. - -| Method | llama.cpp type | Over ggml-rpc | With vLLM PP | Why | -| --- | --- | --- | --- | --- | -| MTP | `draft-mtp` | works; not documented upstream, but run this way on this fleet's split head with acceptance measured ([[Qwen3.8 PP Deployment]]) | no | Context-backed drafter: it needs a draft context and runs in the head process, so RPC devices are simply more devices under it | -| DFlash | `draft-dflash` | same gate, same status as MTP | no | Same draft-context gate (`ctx_dft`); the drafter's KV lands on whichever device the draft context is built on — the one thing to check on a split | -| ngram-mod | `ngram-mod` | safest of the four: no draft model, no draft context, no remote draft pass | no | Draftless — proposals come from cached token n-grams in the head, so the split adds no device work | -| DSpark | `draft-dspark` | **unproven over RPC** — not a missing feature | no | `draft-dspark` exists upstream; nobody has measured it with the target split across two machines | - -**Correction (2026-10-19).** A retired stub in this directory asserted "DSpark -is vLLM-only." False. `draft-dspark` is in upstream llama.cpp — parsed in -`common/arg.cpp`, mapped to `COMMON_SPECULATIVE_TYPE_DRAFT_DSPARK` in -`common/speculative.cpp`, where it carries a Markov head and an anchor-first -block layout and aborts with `DSpark draft has no confidence head: please set ---spec-draft-p-min 0` when the drafter has no confidence head. What is unproven -is running it over RPC, and that is the honest limit of the claim. - -The vLLM column is uniform because that statement is blanket: it names pipeline -parallelism, not a method, so n-gram and suffix are excluded as written. vLLM -gives no mechanism with it; "the drafter must sit next to the target's layers, -which a stage split separates" is a reading, not upstream's words. - -Upstream likewise documents neither for nor against `--rpc` plus -`--spec-type`. What the implementation does say is structural: every -speculative implementation is built in the head process against the head's -device set, and draftless types need no draft context at all. Routing draft -passes to a remote worker, however, has never been tested. - -## Mixed-vendor reality - -What works today: - -- **One vendor per node.** RPC is backend-agnostic by construction: upstream's - own diagram has a host exposing CUDA0+CUDA1, a host exposing Metal, and a - host exposing CUDA0+CPU, all driven by one main host over TCP. Each node - builds its own backend plus `-DGGML_RPC=ON`; nothing speaks CUDA-to-ROCm. -- **Identical build tag on every node**, worker restarted before head. The - verified version-skew gotcha is [[Qwen3.8 PP Deployment]]'s. -- **Speculation on the head** while the split carries the target (table above). -- **RDMA when both peers have it**, auto-negotiated at the handshake; the size - of the gain is third-party and unverified here. - -Unproven or broken: - -- **One binary holding both CUDA and ROCm for a same-host layer split**: - reported to compile and then list only CUDA devices (issue #11506, - third-party, unverified). Vulkan is the reported single-binary cross-vendor - path, and single-host placement is [[Llama Cpp Multi-Device Splits]]'s, whose - verdict is that per-tensor placement across a vendor mix never works. -- **vLLM across a CUDA node and a ROCm node**: no known experiment; both ends - run the same backend (recorded 2026-08-14). AMD's own two-Halo clustering - guide is tensor parallelism over RCCL, not a mixed-vendor split. -- **DSpark over RPC**: unproven, see the correction above. -- **The exact NVIDIA-desktop + Strix-Halo-NUC pair has no published - benchmark**, and the third-party mixed-vendor figures it is estimated from - (Metal+CUDA over 10 GbE, two Strix Halos over USB4, a three-card CUDA+ROCm set - over 1 GbE, a Vulkan triple in one box, an AMD four-node Strix cluster) live on - [[Expected Performance]] and [[Ethernet Bandwidth Math]], each named with its - model and hardware. None was re-verified in this pass. - -## Security - -`ggml-rpc-server` has **no authentication and no encryption**. Upstream's own -warning: the backend is "in a proof-of-concept development stage. As such, the -functionality is fragile and insecure. Never run the RPC server on an open -network or in a sensitive environment!" It binds `127.0.0.1:50052` by default — -safe by accident. Bind a routable address and any host that can reach the port -can allocate memory there and supply the tensors it computes on. Trusted LAN -only, firewalled or tunnelled; never a public address. - -## Unverified in this pass - -Every item was first recorded 2026-08-14, in the research that preceded this -page. Nothing is silently deleted; none is backed by the sources cited below. - -- **RPC protocol version and handshake rule** — recorded as protocol `5.0.0` - with "major must match, server minor must be ≥ client". The README's own - banner prints `Starting RPC server v3.0.0`; the conflict is unresolved. -- **RPC internals** — a hard cap of 16 servers, synchronous graph compute, no - tensor copy between two RPC servers, workers loading tensors on first use, a - hash-keyed cache above roughly 10 MB, asserts on quantized tensors whose rows - are not 512-aligned, and `-rpc` on `llama-bench` (the README names only - `llama-cli` and `llama-server`). -- **vLLM flag surface** — `--pipeline-parallel-size`, `--tensor-parallel-size`, - `--distributed-executor-backend {ray,mp}`, `--nnodes`, `--node-rank`, - `--master-addr`, `--headless`, the `ray start --head` join, - `NCCL_SOCKET_IFNAME`. -- **vLLM mechanics** — one activation handoff per stage boundary per forward - pass, a worked ~512 MB handoff for a hidden-5120 model at batch 8 and 2048 - positions, `--max-model-len` unchanged by PP, ~50% idle at pp=2 batch one. -- **vLLM throughput verdicts** for a 27B across two nodes over gigabit-class - links; superseded as an estimate by [[Expected Performance]]. -- **Mixed-vendor claims** — a CUDA+HIP build listing only CUDA devices - (#11506), Vulkan as the universal cross-vendor backend, AMD's two-Halo - clustering guide being RCCL tensor parallelism, WiFi costing about ten times - wired, the slowest hop setting the decode pace, RDMA worth a few percent. -- **Draft over RPC** — that the draft model stays on the head, and that routing - draft passes to a remote worker has never been confirmed. - -## Citations - -- [llama.cpp RPC README](https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md), accessed 2026-10-19 — server usage and device exposure, `-DGGML_RPC=ON` on both sides, the `--rpc host:port` list, proportional weight+KV split and `--tensor-split`, the `-c` local cache, RDMA negotiation and `GGML_RPC_NO_RDMA`, the proof-of-concept and no-authentication warning. -- [llama.cpp multi-GPU guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md), accessed 2026-10-19 — split modes (`-sm layer` = pipeline parallelism, KV with the layer owner), `-ts`/`-ngl`/`-ctk`/`-ctv`/`-fit`, the pipeline-versus-tensor summary, the tensor-mode limits. -- [vLLM speculative decoding docs, v0.25.1](https://docs.vllm.ai/en/v0.25.1/features/speculative_decoding/), accessed 2026-10-19 — the pipeline-parallelism incompatibility statement, the draft-model version floor, the method catalog. -- [llama.cpp `common/speculative.cpp`](https://github.com/ggml-org/llama.cpp/blob/master/common/speculative.cpp), accessed 2026-10-19 — the speculative implementation set including `draft-dspark`, its draft-context gate, its confidence-head error. -- vLLM issue #6911, "[Feature]: Combine pipeline parallelism with speculative decoding", opened 2024-07-29, closed not-planned 2025-03-28, accessed 2026-10-19. -- vLLM issue #10615, "[Usage]: Does speculative decoding support pipeline parallelism?", opened 2024-11-25, closed not-planned 2025-04-18, accessed 2026-10-19. -- vLLM issue #36643, "[Bug]: Qwen3.5 does not work with pipeline parallelism", opened 2026-03-10, closed not-planned 2026-09-24, accessed 2026-10-19. diff --git a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Sample Script Set.md b/serverAppConfigs-docs/LLM/Pipeline Parallelism/Sample Script Set.md index 24313f6..37a1377 100644 --- a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Sample Script Set.md +++ b/serverAppConfigs-docs/LLM/Pipeline Parallelism/Sample Script Set.md @@ -255,7 +255,7 @@ is 30 GiB. ## See Also -- [[Multi-Node Runtimes]] +- [[Qwen3.8 PP Deployment]] - [[Expected Performance]] - [[Ethernet Bandwidth Math]] - [[DFlash DSpark Speculative Decoding]] \ No newline at end of file From 52dbe18486ac0630aa0e51ebc8ee0f403a1eacf3 Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 19 Oct 2026 10:17:52 -0700 Subject: [PATCH 09/12] docs(vault): the census was lying; corrected, and the honest result is flat docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `scripts/port-census.py` built its corpus from the new directory, so each page under test was held by itself: every era page reported 0 orphan lines. With the corpus excluding the file under test, each candidate holds 14-63 sole-record lines, so no era page is padding and none was trashed. Deletions of restatement only: - Multi-Node Runtimes.md (-257): restated launch material owned by Qwen3.8 PP Deployment.md; index rows and two Related bullets repointed. - log.md: 66 headings folded into 35 dated entries (one entry per date, newest first), titled headings kept as `date — Title` per the spec's wording, and 78 lines of transient deploy chatter dropped where every fact token in the line is held by an owner page (preserved in .AGENTTRASH/20261019-log-transient/). - Authentic: the 5-page split reverted (-93); the parent carries `okf-exempt: length` and the three stub pages stay deleted. check-okf.py accepts `## YYYY-MM-DD — Title`; port-census.py excludes the file under test and gained a docstring note about the trap. One-off compaction tool moved to ~/.AGENTTRASH. Full suite: 451 tests pass, 0 OKF violations, 155 pages, 0 unresolved links. --- scripts/check-okf.py | 11 +- scripts/port-census.py | 32 +++-- .../LLM/Pipeline Parallelism/index.md | 1 - serverAppConfigs-docs/index.md | 1 - serverAppConfigs-docs/log.md | 128 +++--------------- 5 files changed, 50 insertions(+), 123 deletions(-) diff --git a/scripts/check-okf.py b/scripts/check-okf.py index f08593e..70262b6 100755 --- a/scripts/check-okf.py +++ b/scripts/check-okf.py @@ -12,7 +12,8 @@ guidance, so "conforms to OKF" is a test and not a claim: TIMESTAMP `timestamp` present, ISO 8601, not in the future LENGTH lines between 1 and 500 inclusive INDEX-FM no index.md may carry frontmatter (index.md is reserved) - LOG-DATE log.md date headings start `## YYYY-MM-DD`, newest first + LOG-DATE log.md date headings are `## YYYY-MM-DD` or `## YYYY-MM-DD — Title`, + one entry per date, newest first INDEX-COVER every index.md links each of its sibling pages; the root index links every page in the bundle INDEX-NUM an index table row restates a fleet quantity — a digit run of @@ -143,8 +144,12 @@ def check_file(path: Path, root: Path, length_exempt=frozenset()): if not DATE_RE.match(head): continue dates.append(head[:10]) - if head != head[:10]: - v.append(("LOG-DATE", rel, f"date heading carries a title: '## {head}'")) + # OKF requires the heading to USE the date format; a title after an + # em-dash is allowed and costs zero lines, so `## 2026-08-17 — Why the + # pair was rebuilt` is conformant and `## 2026-08-17Rebuilt` is not. + if head != head[:10] and not re.match(r"^\d{4}-\d{2}-\d{2}\s+—\s+\S", head): + v.append(("LOG-DATE", rel, + f"date heading must be the date alone or `date — Title`: '## {head}'")) if len(dates) != len(set(dates)): v.append(("LOG-DATE", rel, "duplicate date heading")) if dates != sorted(dates, reverse=True): diff --git a/scripts/port-census.py b/scripts/port-census.py index 3c38846..aac2b8e 100755 --- a/scripts/port-census.py +++ b/scripts/port-census.py @@ -50,23 +50,39 @@ def held(tok, corpus): def main(argv): new_dir = pathlib.Path(argv[1]) - corpus = [ - p.read_text(errors="replace") - for p in new_dir.rglob("*.md") - if ".AGENTTRASH" not in str(p) and ".obsidian" not in str(p) - ] - blob = "\n".join(corpus) - lost = 0 - checked = 0 + + def corpus_excluding(exclude): + """Every vault page except the one under test. + + The corpus is what must still hold a fact AFTER the file is gone, so a + file that lives inside the scanned directory has to be removed from it — + otherwise every token is "held" by the file itself and the census proves + nothing (it reports 0 losses for every page, which is the bug this + guards). + """ + out = [] + for p in new_dir.rglob("*.md"): + s = str(p) + if ".AGENTTRASH" in s or ".obsidian" in s: + continue + if exclude is not None and p.resolve() == exclude: + continue + out.append(p.read_text(errors="replace")) + return out + + lost = checked = 0 for spec in argv[2:]: if spec.startswith("git-"): ref, path = spec[4:].split(":", 1) old = subprocess.run(["git", "show", f"{ref}:{path}"], capture_output=True, text=True).stdout label = f"{ref}:{path}" + exclude = None else: old = pathlib.Path(spec).read_text(errors="replace") label = spec + exclude = pathlib.Path(spec).resolve() + blob = "\n".join(corpus_excluding(exclude)) for line in old.splitlines(): s = line.strip().strip("->*|# ").strip() if len(s) < 30 or s.startswith("```"): diff --git a/serverAppConfigs-docs/LLM/Pipeline Parallelism/index.md b/serverAppConfigs-docs/LLM/Pipeline Parallelism/index.md index 9e7f00f..1ffc51b 100644 --- a/serverAppConfigs-docs/LLM/Pipeline Parallelism/index.md +++ b/serverAppConfigs-docs/LLM/Pipeline Parallelism/index.md @@ -69,4 +69,3 @@ graph LR | Page | Owns | |---|---| -| [[Multi-Node Runtimes]] | How each two-machine runtime is built and launched, what it can combine with, and the mixed-vendor verdicts. | diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 439a818..03c44b9 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -169,7 +169,6 @@ is a locator: one row, one clause, no numbers. |---|---| | [[Ethernet Bandwidth Math]] | Bytes-per-token math for pipeline versus tensor parallelism, and the link budgets that follow from it. | | [[Expected Performance]] | Estimated split-model throughput over ethernet, with the benchmark anchors the estimates rest on. | -| [[Multi-Node Runtimes]] | How each two-machine runtime is built and launched, what it can combine with, and the mixed-vendor verdicts. | | [[Qwen3.8 PP Deployment]] | The real split deployment: which node is head and which is worker, the flags, and the verified gotchas. | | [[Sample Script Set]] | An educational two-node RPC split script set, for reading only — never run it or deploy it. | diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index c415b6c..8974105 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -53,6 +53,23 @@ stale the moment main moves; re-probe `/slots` and the running cmdline before writing the number, not after.** +- **A census that includes the file under test proves nothing.** `scripts/port-census.py` + takes the new directory as its corpus, so every era page checked against + `serverAppConfigs-docs` was trivially held by *itself*: ten pages reported + `0 unheld tokens`, including `kingjones-q8agent-rocmfpx` at 481 lines whose own load + ledger cannot live anywhere else. Real numbers after the corpus excluded the page under + test: **14–63 sole-record lines per page — no era page in this vault is padding**, each + the only copy of a campaign ledger, which is what *Documentation Policy* protects. + **Law: a tool's own docstring promise ("excluding it") is not a check; assert the file + is absent from the corpus before believing a zero.** Same shape as *Key Lessons* → "A + scout verdict is a lead, never a finding": a mechanical tool is a scout too. +- So "compact the docs" reduces to deleting **restatement**, which is what landed: + `Multi-Node Runtimes.md` (−257, it restated launch material owned by + [[Qwen3.8 PP Deployment]]), 78 lines of transient deploy chatter out of this log (dropped + only where every fact token in the line is held by an owner page, preserved under + `.AGENTTRASH/20261019-log-transient/`), and 148 lines out of `AGENTS.md`. Docs flat, + operator doc down; the growth is the two enforcement tools. + ## 2026-10-18 - **frank-zealot resliced to two windows of exact KV (Frank's call, PR #21).** `--ctx-size 786432 --parallel 3` at q8_0 K+V → **`524288 / 2` at f16 K+V**, so @@ -79,7 +96,6 @@ **issue #3 research: AMD PerfOpt retires the reason for `amd_iommu=off`** - - Ask: does a recent kernel change remove the IOMMU translation penalty that justifies the 2026-09-11 `amd_iommu=off` boot line, i.e. can the XDNA2 NPU come back? Answered and documented; **no host change** at the @@ -113,11 +129,9 @@ `frank-npu.md` § *Why it is retired* corrected — the "`pt` is not a middle ground" law is dead — the AGENTS.md key-lesson bullet re-keyed, and a 4-step maintenance-window procedure that measures locally instead of - inheriting the vendor number. Nothing deployed — no service was touched. **boot line flipped: Dankclaw now boots `iommu=pt`** - - Window executed 01:24–01:35 on Frank's go-ahead, with the A/B the research demanded. Provenance first: `/etc/default/grub` had never been in the repo while the IOMMU line was the most consequential kernel decision @@ -125,7 +139,6 @@ byte-exact) and moves only through `scripts/set-dankclaw-bootline.sh` — diff, install 0644 root:root, `update-grub`, **verify the default `grub.cfg` entry**, reboot on `--reboot` only. The deployer plans nothing - under `dankclaw/host/`: kernel state is not a container restart. - **Verdict: identity DMA costs nothing on this host.** Byte-identical traffic, +2.0 % cold prefill going OFF→PT; the 13–16 % that justified `amd_iommu=off` for five weeks never existed here. Decode's apparent @@ -133,7 +146,6 @@ same boot spread 31.8–47.8 t/s. - **The XDNA2 NPU is back:** `amdxdna` bound `0000:c6:00.1`, `/dev/accel/accel0` `crw-rw---- root:render 261,0` with `dank` in - `render`. No endpoint deployed — revival needs a quadlet with `AddDevice=/dev/accel/accel0` + `GroupAdd=keep-groups`. - Deliberately NOT done: the `pp_dpm_fclk` level-7 pin (it changes iGPU timing for every workload and nothing co-runs on the NPU yet) and any @@ -142,7 +154,6 @@ - Gates after boot: 0 failed units, 32 IOMMU groups (31 identity + 1 DMA = the Genesys Logic SD reader, a kernel quirk), interrupt remapping on; frank-dragoon's ledger byte-for-byte the same shape and Gitea / Caddy - edge / Grafana answered 200. New tool `scripts/stage-frank-dragoon-eval.sh` rebuilds the A/B corpus byte-identically — run it through `bash -lc`, `uv` is in `~/.local/bin`. Docs re-keyed: the decision record `dankclaw-iommu-translation-off.md` (new § *execution*), `frank-npu.md` @@ -171,15 +182,11 @@ `SPEC_TYPE=draft-mtp` drop the flag. And `up{job="frank-zealot"}` had read 0 for the **entire retained Prometheus history**: llama.cpp answers **501** on `/metrics` unless `--metrics` is passed, so the scrape failed - beside a healthy `/health` and `LlmEndpointDown` fired for days. `METRICS=1` is now the default and the seam pins it. -- Deployed 13:16 (`deploy.sh dankclaw --service frank-zealot`) — and the first boot was **not the shipped payload**: a leftover `frank-zealot.service.d/ptq1.conf` from the bench window pointed `FRANK_ZEALOT_ENV_FILE` at `/tmp/zealot-arm-current.env`, so the unit - booted that arm. `/health` 200, `/v1/models` echoed the alias, Playwright 33/33 — **nothing in the gate reads the payload.** Drop-in moved to - `~/.AGENTTRASH/`, `daemon-reload`, restart: the shipped trio booted. (The drop-in was not a rogue — see the PTQ1_0 entry below.) - Measurement law: the first battery reported 117 and 72 t/s because it timed natural generations that stopped at 24 tokens — wall-clock aggregate @@ -218,7 +225,6 @@ **frank-dragoon: the 401,408-position pool goes live; docs re-keyed to the measured boot** - - `./scripts/deploy.sh dankclaw --service frank-dragoon` — e2e 33/33, and the drop-in check the zealot incident bought ran FIRST: `DropInPaths` empty, so the repo env governed. 2026-10-18 — frank-dragoon: the KV pool @@ -232,7 +238,6 @@ page cache loses. - Every document quoted **~10.8 GiB**, the ~28 KiB/position arithmetic; the engine reserves **11.0 GiB** and its preflight bills **11.3**. The measured - figure wins, and every "committed but not yet deployed" clause was retired with the measurement (AGENTS.md, the env lore block, both index pages, the iommu page, the runbook). - PTQ1_0 provenance corrected (see the dedicated entry): the arm that @@ -243,7 +248,6 @@ MiB freed, then scratched by Frank's 03:33 order in favour of the 9B. Nothing is owed on it. - Census of the halogen HTTP surface while doing it, because the fleet bench - law quoted the wrong engine: frank-dragoon answers `/health`, `/metrics`, `/v1/models`, `/v1/chat/completions`(POST) and 404s `/ping`, `/props`, `/completion`, `/tokenize`; the chat response carries **no `Server-Timing`** header. So the `/completion` `timings` object and the @@ -251,15 +255,12 @@ dragoon's. AGENTS.md's Benchmarking Policy re-keyed to say which engine owns which instrument. -**frank-tts phase 1: the cloned-voice endpoint, named and tested (not deployed)** - - Voice-over goal, phase 1 (Gitea issue #2). New service **frank-tts** on Dankclaw: `dankclaw/voiceservice/` (stdlib-only `app.py` + `Containerfile` + rootless quadlet), `scripts/frank-tts.env`, `tests/frank_tts_unit.bats`. One OpenAI-shaped `POST /v1/audio/speech` surface; `voice=` is honoured only when `/voices//reference.wav` exists, otherwise a 400 listing the voices that are — **no silent fallback voice**; oversized - input is a 414, not a half-narrated sentence; `/health` reports backend + voice list; no audio ever reaches the log. - **Naming law applied (Frank's call):** treat it as a first-class model endpoint → alias-named `frank-tts` (TTS endpoint, so not `frank-asr`), @@ -309,7 +310,6 @@ only and a bridge container cannot reach a host `127.0.0.1` bind — the lesson that 502'd hermes — so `host.containers.internal:8880`, which `scripts/frank-tts.env` documents, is correct only for a routable bind. - The deployed pair shares one quadlet network and the façade addresses the engine by container name. - **PR #12** (`feat/open-notebook-podcast` @ `114dd7a`) is **merged** as `1eb8b21` (Frank's OK): open-notebook + SurrealDB quadlets on the private @@ -319,9 +319,6 @@ runnable: nothing listens on 7772 until the voice stack lands, and the episode destination stays interim pending Gitea #7. -**open-notebook: the podcast generator's config lands (nothing deployed)** - - - Goal: **open-notebook** as the fleet's podcast generator on Dankclaw — notes and sources in, a multi-speaker episode out, script written by **frank-zealot**, voices rendered by **frank-tts**, no cloud provider @@ -332,7 +329,6 @@ frank-dragoon's pinning style, because `:latest` moving underneath a rootless service is how you get an unbootable pod after an unrelated `systemctl-daemon-reload`. Both health probes are argument-only - (`curl -fsS …/health`, `/surreal is-ready`), and `is-ready` refuses to pass without `SURREAL_PASS` in the environment — a credential that must never ride the command line, so `Exec=` carries flags only. - **Host 5055 was already spoken for** — seerr has held `127.0.0.1:5055` — @@ -369,20 +365,16 @@ `podman unshare`. One mount line; one deletion undoes it. - **Two UI buttons will fail, and that is cosmetic:** Test Connection does `GET /models` and the TTS probe synthesises voice `alloy`; - frank-tts serves `/health` and `/v1/audio/speech` only. Model rows are created *by name* and the voice field is free text, so `narrator1/2/3` is typed in and never discovered. Documented, not worked around. - Three deployer seams cost time and are now law-shaped: a service must be registered in **five** places — `SERVICE_ORDER`, `DANKCLAW_SERVICES`, the - `deploy-remote.sh` dispatch case, its pre-deploy requirement, and `deploy.sh`'s dispatch regex, where an omission silently plans an empty set for `--only`; a `dankclaw//*` case must precede the `dankclaw/*.container` glob because that `*` crosses `/`; and the changed-path routing lives *inside* the per-path loop, after the `scripts/*.env)` case, not at the top of the file. -**frank-tts-engine: the arm A engine becomes deployable (nothing deployed)** - - Goal: turn the engine the day's A/B picked into a **repo-owned, boot-persistent, self-healing container** — `frank-tts-engine`, `qwentts.cpp` commit `5916bf7c79dbb4e66fc059cdc507a79a23a28bb4`, resident @@ -395,7 +387,6 @@ containers therefore sit on one committed quadlet network, `frank-tts-engine.network` at **10.89.5.0/24** (the next free /24 after open-notebook's 10.89.4.0/24), the façade addresses the engine by - container name, and diagnostics — including the deployer's health gate (`wait_container_healthy`, there is no URL to dial) — run through `podman exec`. - **The boot warmer is a unit hook, not an operator habit.** The registry is @@ -403,7 +394,6 @@ reference, so `ExecStartPost=-/usr/bin/podman exec frank-tts-engine /app/warm-voices.sh` with `TimeoutStartSec=900`; the `-` prefix is what stops a failed warm from crash-looping the endpoint under - `Restart=always`, and it logs `cold=` and `warm=` per voice on every boot. - **Thread law recorded before anyone tunes it away:** no `--threads` exists; `src/backend.h` uses `hardware_concurrency()/2`, so a cpuset pin silently changes the thread count. Nothing pins this container. @@ -417,7 +407,6 @@ **frank-tts: the 7772 endpoint gets its deploy wiring, and voice naming is written down** - - Stage 2 of the voice stack: the **façade** in front of the engine is now a deployable service too. Runbook: [[Frank TTS]]. - **Two attached networks, in a meaningful order.** `dankclaw_default` @@ -446,14 +435,12 @@ - **No Prometheus job, on purpose.** The façade exports no `/metrics`, Prometheus 3 refuses HTML/JSON content types (the `comfyui` and `services` probes are documented down for exactly that), and `LlmEndpointDown` is - scoped `up{job=~"frank-.*"}` — so a `/health` scrape would page on Signal every few minutes for a healthy endpoint. That is the frank-archon never-green lesson pointed the other way, written in the page instead of quietly ignored. **voice-pack builder: the 10-second source-material gate becomes code** - - Third slice of the voice stack: `scripts/build-voice-pack.sh` turns a source recording plus a crop offset into a deployable pack — `reference.wav` (24 kHz mono `pcm_s16le`, 18–25 s), the engine's own @@ -505,18 +492,14 @@ skips `podman build` on a matching `cfhash` label, and that hash covered the `Containerfile` alone — so editing `app.py` or `tts-engine-warm.sh` (both `COPY`'d) skipped the build while the deploy reported success and - `/health` answered with old code. Each case arm now hashes the files its `Containerfile` copies. - Docs carry the lessons where an operator will hit them: a **Locally built images** section in [[deployment]], two new troubleshooting lines in [[Frank TTS Engine]] (all-voices-cold = quoted `WARM_TEXT`; façade 502 beside 200 = model loaded ≠ voices registered), and `Voice Packs.md` step 4 corrected — the warm hook re-registers **every** pack on every run, - because the engine replaces a name it already holds, so a restart is what an engine bump needs and not what a rebuilt pack needs. -**frank-tts-engine deployed: it builds, it serves, and the warm hook died on the kernel's argument cap** - - Merged as **#14** (`747c665`) and deployed `--service frank-tts-engine`; the image built clean — the first real proof of the `useradd` fix — and the unit was `healthy` after 31 s. @@ -526,7 +509,6 @@ a single argument at 131,072 bytes (`MAX_ARG_STRLEN`) and a shipped ~955 KB `reference.wav` base64's past that, so curl never executed and the hook logged that as "HTTP no response". Fourth silent fault of the voice stack, - same species as the other three: `/health` stayed green throughout. Fix: the body goes to a `mktemp` file and curl gets `@file`; the regression test is the honest shape — real listener, real `curl`, a 300 KB fixture pack — so it fails with the same kernel error the host produced. Every shipped @@ -552,7 +534,6 @@ - Merged **#15** (`0315a62`, the warmer's argument-cap fix) and **#16** (`d2ce669`, the comment sweep), then deployed both voice services in order (`--service frank-tts-engine`, then `--service frank-tts`); Playwright gate - 33/33 twice. Deployed-state proof from a LAN client: `/health` → `omni-http` with all three voices, and the same sentence in `narrator1` vs `narrator2` came back with different sha256 and different RMS — the name actually selects a voice. @@ -564,7 +545,6 @@ errors instead of degrading; and `models.yml` loads at process start while `config.yml` is read live, so a session older than the `frank-tts` row fails exactly like a dead endpoint. -- Corrected the same evening, by breaking it in a restarted session: a harness call that names no voice reaches 7772 with **no `voice` key at all** — `400 unknown voice None; available voices: …` — because no setting feeds one. The endpoint therefore grew `DEFAULT_VOICE` (its own PR): a @@ -615,7 +595,6 @@ - Also in the window: **PR #22 merged** (`0c7fe8b`, `dank/command-center`) so `agent/extensions/dankvoice.ts` landed on the auto-load path — proven in a plain `omp` session with no flags: `/say merged build check` → WAV to the - player, temp dir cleaned. Sessions open before the merge need a restart to pick the command up. **frank-zealot: 2 × 262,144 over unquantized f16 KV** @@ -638,7 +617,6 @@ ## 2026-10-17 **frank-zealot: Bonsai-2-27B PQ2_0 trial + cutover (2 × 200,704, q8_0 KV)** - - 2026-10-17 — frank-zealot: **Ternary-Bonsai-2-27B (PQ2_0) cutover**, two slots at 200,704 over q8_0 K + q8_0 V on the RTX 3090, fetched by `scripts/download-frank-zealot-bonsai2.sh` (exact bytes + sha256, @@ -672,12 +650,10 @@ `serve-frank-zealot.sh` still DEFAULTED to the retired payload's paths (a defaulted run would have served the old model — now re-keyed to the shipped trio), and the OpenWebUI `model` row's `meta.description` is live - DB state (re-keyed with a DB backup; no `open-webui` restart forced — forcing it logs every family session out for one cosmetic string). **dankserver disk purge: Swift w4dn8 rollback + orphan caches (docs re-key)** - - Ask: "dump the swift rollback and the GLM GGUF — make sure the model caches and agenttrash are empty as well." Preflight first: `lsof` empty on both payloads, the serving trio names only the live path, and a podman @@ -689,7 +665,6 @@ rollback is now a **re-download, not a local file**. Corpora (grams, kld-ref, calibration JSONL) deliberately KEPT: they are the re-convert and KLD-gate inputs. -- Post-checks: frank-archon active and `/health` ok, serving payload byte-size intact, the live kvcache tier untouched, no process references a deleted path. - Docs re-keyed: AGENTS.md fleet row + frank-archon section + both runbook @@ -701,7 +676,6 @@ **frank-dragoon cutover: native v2 abliterated payload, card's memory-saving pool** - - 2026-10-17 — frank-dragoon: cutover to the native v2 abliterated halogen payload with the card's memory-saving KV pool; census first: the n-gram table was ALREADY disk-backed and the only real lever was the KV pool. @@ -733,7 +707,6 @@ **frank-dragoon: ceiling and pool cut to 200,704; crawl reset fix** - - 2026-10-17 — frank-dragoon: context ceiling and KV pool cut to 200,704 (49 × 4096, the per-slot rung frank-zealot runs; nothing in the fleet needed the bigger window). See [[frank-dragoon]]. @@ -747,7 +720,6 @@ - Probes on the live endpoint: over the ceiling → `400 … tokens over` in 0.86 s — preflight rejects before any prefill, so probing costs the endpoint nothing. -- Deploy verdict: the endpoint dispatched healthy (`/health` 200), but the `[3/3]` Playwright gate failed on `12-crawl` and the deployer correctly said **DEPLOY INCOMPLETE**. Root cause was NOT the endpoint — it was Plex Web, signed out in the `qa-e2e` profile, running its own sign-in flow, @@ -766,7 +738,6 @@ ## 2026-10-15 **frank-archon: publisher iq4r payload swap (MODEL_FILE-only)** - - 2026-10-15 — frank-archon: publisher iq4r payload swap, `MODEL_FILE`-only — `StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe`, the engine author's published rad-convert of the OFFICIAL Qwen3.8-Flash-Next, **no other @@ -779,19 +750,15 @@ - Fetch: new `scripts/download-frank-archon-iq4r.sh` (hfget, exact bytes + Hub LFS sha256, `--verify-only` default) as a transient unit. The cutover **self-severed by design** — a detached `deploy.sh` survived the session - death when frank-archon restarted; engine healthy in ~100 s, and the prefix tier RECOVERED across the swap (arch+tokenizer unchanged). - e2e gate went red on three dankclaw specs — NOT the swap: dankclaw rebooted 01:41 and the gluetun boot-race dependency-failed `searxng`, `marreta`, and `marreta-browser` (jobs consumed while gluetun's start job - retried; `Restart=always` never armed because the services never ran). - Cascade restarted; skill codified (`gluetun-dependent-boot-race-recovery`). frank-archon's own health gate passed every deploy. **authentik: two more human accounts (Dankclaw)** - - **Live — mitch (pk 22) and eric (pk 23) provisioned via `ak shell` ORM, no repo objects**, each `@franksdank.cloud` in the human-user template shape (password via `set_password` on stdin only — never argv, @@ -805,8 +772,6 @@ power failure on dankserver reset `/sys/module/ttm/parameters/pages_limit` from the manual 200 GiB raise back to its default (half of RAM = 125,988 MiB), below the ~147 GiB pinned-RAM ledger the engine needs; the serve script's - preflight named the refusal and exited 1, and `Restart=always` re-ran it - **6,919 times in ~20 h** — `activating (auto-restart)`, exit 1 in 50 ms, no container, no engine log. Law adopted: **a sysfs raise is a countdown to the next reboot**, so the gate raises the knob itself. New `scripts/ttm-preflight.sh` is sourced by the serve script (verify, raise, @@ -879,7 +844,6 @@ (4,379 t/s), so the cliff is real and sits at ~26 GiB. The 2-slot arm that would clear it (pool 3,072 + slab 27,897) remains untested — and the paragraph above is amended: that arm's payoff is +22 % prefill at 2 slots, - not the +8.9 % the deployed pair ever showed (24,000 → 25,600 = 0 %). - **Frank's decode challenge, answered with numbers**: 512 tokens, greedy, cold — **decode 99.0 t/s single-stream at 43.5k and 93.3 at 197k** (decode is nearly window-free: −6 % for 4.5× depth), while **3 concurrent streams @@ -893,10 +857,7 @@ tokens, per-stream + aggregate) so no future shape change can be priced on prefill alone. -## 2026-10-11 -**frank-archon: unquantized (bf16) KV at 2 × 200,000** - - +## 2026-10-11 — frank-archon: unquantized (bf16) KV at 2 × 200,000 - 2026-10-11 — frank-archon: **unquantized (bf16) KV at 2 × 200,000** on exact attention state, with pool, slab and host pool rebalanced around the bigger cache. See [[frank-archon]]. @@ -941,7 +902,6 @@ `frank-archon` active/enabled (container `frank-archon`, :7770), `frank-dragoon` active/enabled (container `frank-dragoon-halogen`, :7770), `frank-zealot` active/enabled (journal shows 87–93 t/s serving - compaction on :7771). `/health` 200 ×3; `/v1/models` reports the three ids; host checkouts grep zero residue; e2e passed 32+1flaky, 32+1flaky, 33; Prometheus `frank-archon`/`frank-dragoon` up. - **Pre-existing gap found by the verification, NOT a rename regression:** @@ -950,7 +910,6 @@ it, so `up{job="frank-secondary"}` had NO series at all in the last 24 h (the zealot metrics + the queue-depth alerts on it have been dead since the Qwen3.5 swap). Fix = one flag in `serve-frank-zealot.sh` + a ~1 min - zealot restart; not applied in the rename window. ## 2026-10-09 - **Live — five authentik human accounts provisioned (Dankclaw, `ak shell` @@ -986,7 +945,6 @@ `webui.db.bak-20261009`): `model` row (`frank-zealot` → `frank-zealot`, params + `meta.description` preserved), the `access_grant` row (`model`/`frank-zealot`/`user`/`*`/`read`), and `openai.api_configs` - `model_ids` on all three backends; `systemctl --user restart open-webui.service`. Family-account proof, in a real browser as `qa-e2e`: the welcome screen selects `frank-zealot` and `POST /api/v1/chat/completions {"model":"frank-zealot"}` answers @@ -997,7 +955,6 @@ - **Live — Hermes re-keyed** (`config.yaml` `frank-dragoon` → `frank-dragoon` in the provider block, `model.default` and the three `dankclaw-large` auxiliary routes; backup `config.yaml.bak-20261009`; `hermes.service` + - `hermes-webui.service` restarted, gateway log clean, `/health` ok) and the repo baseline `dankclaw/hermes/config.yaml.example` synced to the live shape (context 262144). **Flagged, not fixed:** Hermes still routes its `dankserver-small` auxiliary tasks at the id `frank-medium` on @@ -1006,13 +963,10 @@ they need a routing decision (archon capacity), not a rename. - **Deployer blind spot found by accident:** every `deploy.sh dankserver --service frank-archon` since 2026-10-08 04:09 - restarted the **staging worktree**, not the deploy checkout — a campaign drop-in `frank-archon.service.d/bench.conf` overrode `Environment=` and - `ExecStart`. The health gate went green each time. Verified parity before removing it (serve script + unit byte-identical, env identical except the alias, staging HEAD an ancestor of `main`), moved the drop-in to `/home/dank/.AGENTTRASH/20261009-frank-archon-bench-dropin/`, and - restarted: `/v1/models` then answered `frank-archon`. Post-deploy behavioural proof is now part of the runbook. - Repo: seams re-pinned (`frank_zealot`/`frank_dragoon`/`frank_archon` unit bats), fleet harnesses retargeted (`bench-probe`, `deep-probe`, @@ -1132,13 +1086,11 @@ (refusals gone), vision back (the v2 tower loads beside the checkpoint), MTP head inside the checkpoint, 2×200,192 slots, `HALOGEN_SPEC_ADAPT=0`. Runbook: `serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-swift15.md`. - **Incident (03:09:39–03:11:42):** the deploy started the unit while host `MemAvailable` was ~74.7 GiB, 3.34 GiB short of the engine's pin-gate requirement (62.11 GiB checkpoint + 16 GiB floor = 78.11 GiB); it crash-looped 8 boots, each dying in ~5 s at the pin gate, and self-resolved in ~2 min when the dying 27B engine freed RAM (boot #9 pinned clean at 88.4 GiB available). **Latent risk + fix:** the host was - left with only ~1.8 GiB `MemAvailable`, so a restart would crash-loop again; `serve-frank-dragoon.sh` now has a host-RAM preflight that checks `MemAvailable` against the pin gate BEFORE `podman run` and WAITS up to `FRANK_DRAGOON_RAM_WAIT` (600 s) for the RAM to clear, refusing loudly if @@ -1160,7 +1112,6 @@ `reasoning_content`; models.yml flipped accordingly). Container renamed to the unit name (Frank's call; `frank-archon-qwen` → legacy list); deployer image gate → `podman image exists` (local builds carry no - RepoDigests) and health gate reads the BODY — FreeToken 200s with `{"status":"loading"}` mid-load, which false-greened a deploy at 6 s (vLLM 503'd; the bare curl was honest there). Verified: FLEET-OK completion, reasoning split (17×24 → "408"), native `tool_calls`, vision @@ -1281,11 +1232,9 @@ 7,653 slots, budget 31.29. - Shape G v1 (0a4ff30) tried util 1.0: **both workers refused init before a single weight loaded** — `ValueError: Free memory on device cuda:N - (31.29/31.86 GiB ...)`, NRestarts=2 crash loop, zero payload progress. The CLI field is `Field(gt=0, le=1)`, but the worker demands free > budget, so budget == total is a trap, not a knife-edge. Frank: ".982 is the law, never 1.00" — now with a second proof. - - Shape G v2 (c67a2ae, deployed green incl. e2e 33/33): util back to 0.982, graph base to 0 (the shape F boot reported a NEGATIVE -0.06 GiB capture delta, so the 256 MiB base was fiction; 0 + 12 MiB/graph still prices a future 18-graph spec-decode shape at 0.21 GiB), KV_BYTES @@ -1305,7 +1254,6 @@ ## 2026-10-02 **frank-dragoon cutover: vLLM MiMo -> ROCmFPX Q8_0_ROCMFPX_AGENT + MTP** - - 2026-10-02 — frank-dragoon: vLLM MiMo → ROCmFPX `Q8_0_ROCMFPX_AGENT` payload + MTP from the repo's own bundled head, one session at the native 262,144. See [[frank-dragoon]]. @@ -1324,7 +1272,6 @@ **frank-zealot: Q5_K_M, then three contexts + vision, sized by two boots** - - 2026-10-02 — frank-zealot: Q5_K_M payload, then three contexts + vision in one day, sized by two boots (`CTX_SIZE=540672 PARALLEL=3`, f16 K + q8_0 V under flash attention, `--mmproj`). See [[frank-zealot]]. @@ -1353,7 +1300,6 @@ **frank-archon shape D: 3 x 200704, fp8, pool sized for 2 contexts** - Frank's order: shrink frank-archon to a 200704-token window with about two resident full-length contexts, cut the KV pool, and drop MNBT to 1024. The first attempt (`30904ba`, 2 x 200704 + SEQS=2) **crash-looped**; the revert @@ -1362,7 +1308,6 @@ top of the proven-good base, root cause first. - **ROOT CAUSE: `--max-num-seqs 2` cannot compile this model. It was never OOM, never MNBT, never the KV dtype.** Four single-variable boot probes - against the then-deployed shape (production stopped for each, candidate booted AS the service): 262144 x SEQS 2 fails at BOTH MNBT rungs with `torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated (L['query_start_loc'].size()[0]) ... specialized it @@ -1398,13 +1343,11 @@ top of the proven-good base, root cause first. - **Harness contract followed the window:** `models.yml` `contextWindow` 262144 -> **200704**; side effect: frank-archon's compaction request fits inside a frank-zealot slot again. A comment-only follow-up `295d578` was - deployed so the per-service revision marker sits at HEAD. Rollback for the pre-shape-D fp8+MTP4 era stays git tag `frank-archon-fp8-mtp-20261001`. Runbook [[frank-archon]]. **plex: the readiness gate probed HTTP against a TLS-only service** - Found by accident, and it had been latent since the secure-connections posture was set: `deploy_plex` gated on `curl -fsS http://127.0.0.1:32400/identity`, which **cannot** succeed. PMS drops plain HTTP off-loopback - the probe's @@ -1428,18 +1371,14 @@ and then answers with nothing, and curl reports "Empty reply from server" (52). got the same treatment. Fixed to `curl -fsSk --max-time 3 https://127.0.0.1:32400/identity` (`-k`: PMS serves its own self-signed cert on 32400). Re-deployed `39f0e0d`: plex recorded deployed, e2e - 33/33, frank-archon untouched (`health=200`, NRestarts=0). - **Operating lesson: `deploy.sh ` with no `--service` is a - fleet-wide drain.** It restarts every service whose recorded revision is behind HEAD and whose paths changed since. To touch one service, name it. **frank-archon shape E: `--expert-precise-memory` (+1,026 arena slots), a mounted compile cache, and the P2P answer** - Shape D left one question open — the plan's overhead term was reserving 2.35 GiB of compile headroom the boot never used — and the answer to Frank's free-VRAM question arrived together with a boot-cost bug that had been costing 181 s on -every restart since the cutover. - **`--expert-precise-memory` is real and it fits (option A).** Boot-probed on the shape-D base with production stopped, the candidate booted AS the service @@ -1463,11 +1402,9 @@ every restart since the cutover. 1.38x` — measured, rejected as the shipped value, kept as the conservative fallback. With 3.15e9/rank the endpoint keeps shape D's 1.99x concurrency and gains the arena, which is the shape D follow-up had actually been looking for. -- **THE COMPILE CACHE WAS BEING THROWN AWAY ON EVERY RESTART — 181 s each.** Measured: a fresh container has **no `/root/.cache` at all** (`No such file or directory`); after our boot it holds `/root/.cache/vllm` at **1,154,714,333 B in 4,665 files** plus `/root/.cache/comgr` at 229 MB, inside - the ephemeral layer, and `podman rm -f` (every deploy, restart, reboot) discards them. The engine names the loss itself: `Compiling a graph for compile range (1, 1024) takes 173.19 s` ... `torch.compile took 181.20 s in total` ... `init engine (profile, create kv cache, warmup model) took 217.26 s @@ -1520,7 +1457,6 @@ every restart since the cutover. not a crash — the container and its serve script must be polled by recorded id and their logs dumped on the exit path, or a failure looks like a bad flag. The flag was fine both times. -- Live: `health=200`, `NRestarts=0`, arena 17.22 GiB / 7,522 slots, KV 398,658 tokens / 1.99x. Docs: [[frank-archon]]. - **Deployed `1ac4d48`, e2e 33/33, and the live plan matched the probe exactly:** `'expert_precise_memory': True`, `runtime overhead … 5.25 GiB`, `expert cache @@ -1529,7 +1465,6 @@ every restart since the cutover. precise overhead and the arena growth are now production numbers, not a probe's projection. - **The compile-cache fix was then proven on the same endpoint, two consecutive - boots of the deployed shape:** cold `Compiling a graph for compile range (1, 1024) takes 179.24 s` → `torch.compile took 188.26 s`; warm `Directly load the compiled graph(s) for compile range (1, 1024) from the cache, took 2.143 s` → `torch.compile took 3.97 s in total` → `init engine (profile, @@ -1560,14 +1495,11 @@ every restart since the cutover. **frank-archon RESTORED to fp8 3 x 262144 (bf16 boot fails)** - bf16 (--kv-cache-dtype auto) at 2 x 262144 did not stay up, with OR without --expert-precise-memory: -- with precise: NRestarts 1->3, each attempt ~6.7 min then `Deactivated successfully` (API server drains after an EngineCore fatal error) — cause not captured, the serve script rm'd the container. -- without precise (default 7.60 GiB overhead, arena 14.3 GiB): NRestarts climbing, container `Exited (1)`, and the journal finally showed the real error: `RuntimeError: Worker failed with error 'Constraints violated @@ -1588,12 +1520,9 @@ Next: bisect bf16 vs SEQS in a window (bf16 + SEQS=3 isolates the dtype). **frank-archon: --expert-precise-memory + bf16 = boot crash loop** - **Reverted.** The two changes Frank asked for were applied together and the -result crash-looped: `NRestarts` climbed 1 -> 2 -> 3, each attempt running ~6.7 minutes then logging `Deactivated successfully` (vLLM's API server drains and exits 0 after an EngineCore fatal error, so `podman wait` returns 0; the -new `Restart=always` turned a silent death into a visible loop). The budget arithmetic explains it without needing the error text, because the arena is DEVICE-resident (the fp8 3-window shape's plan predicts 28.27 GiB and @@ -1621,7 +1550,6 @@ by paying for it elsewhere (smaller pool, fewer sessions, or a shorter context). **frank-archon: 2 bf16 windows + --expert-precise-memory** - Frank's final shape for the night. Two changes at once, both measured in principle before the boot: @@ -1647,7 +1575,6 @@ planned arena; it comes out of free VRAM instead. **frank-dragoon: Q4_K_XL payload + 4 x 163840 sessions** - Frank's call. Payload swapped from ajgazin's **abliterated** Swift-1.5 UD-Q8_K_XL (29.3 GiB) to **unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL** (16.35 GiB, sha256 3f227079…b01e, vanilla/Apache-2.0 — the endpoint is no longer @@ -1669,7 +1596,6 @@ weights 16.35 + KV 40.0 + drafter 1.07 + linear state 0.56 + runtime ~4 = **e2e gate flakiness (12-crawl) — do not panic at the banner** - `specs/12-crawl.spec.ts:23` ("every user open link reaches its app") failed the deploy gate twice and then passed cleanly on a manual re-run: - run 1 (deploy): `Expected "plex.franksdank.cloud"` / `Received "app.plex.tv"` @@ -1688,7 +1614,6 @@ any repo service. **frank-archon bf16 no-spec cutover** - - `/scripts/frank-archon.env` — bf16 no-spec cutover, FINAL shape after the night ladder: MAXLEN **241664** x SEQS 3, KV_BYTES **10,184,000,000**/rank, KV_DTYPE auto (bf16), MNBT 1024, GATHER CHUNKS=32/LANES=64, UTIL 0.98; @@ -1725,7 +1650,6 @@ any repo service. - **Client contract fixed (thinking + tools).** The live endpoint returned a null `reasoning` field and dumped the thinking text INLINE in `content` with a dangling `` (measured: `'We need answer…Brief.\n\n\n391'`) - — the malformed thinking block clients choke on — because the deployed argv had no `--reasoning-parser`. The GLM rollback (`d91379d`) had dropped all three contract flags and the qwen rollback inherited the gap. Restored from `ecaad36`'s values, confirmed against this fork's registries: @@ -1755,11 +1679,9 @@ any repo service. knife-edge shape change, send a large-`max_tokens` request before calling it done. FOUR slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93 GiB per rank for activations/graph capture. Frank's call 2026-10-01. -- `/scripts/frank-archon.service` — **Restart=always, not on-failure** (plus `StartLimitIntervalSec=0` and `OOMScoreAdjust=-500`). A runtime worker fault makes vLLM's API server exit CLEAN, so ExecStart returned 0, systemd logged "Deactivated successfully", and on-failure NEVER fired: the endpoint stayed - down until a human noticed. An explicit `systemctl stop` still wins, so bench windows and deploys are unaffected. - **Final shape (Frank's call, 2026-10-01): 3 x 262144, pool at the FLOOR, MNBT 2048, rest to the arena.** MAXLEN 262144 x SEQS 3 = 786,432 tokens of @@ -1779,7 +1701,6 @@ any repo service. **frank-dragoon cutover: gufo -> vLLM TheRock, MiMo-V2.6 BF16 + cross-model MTP** - - **Question that started it:** can a Qwen3.5 MTP drafter attach to frank-zealot, and how does vLLM run the FULL BF16 MiMo checkpoint? Answer 1: not on frank-zealot — SGLang reads the MTP head from the TARGET checkpoint @@ -1793,7 +1714,6 @@ any repo service. prefill / 512 decode):** base 17,876 t/s prefill / 13.1 t/s decode (13.1 is ~94% of the LPDDR5x BF16 roof - decode here is bandwidth, not a bug); +DFlash-8 7,976 / 17.4 (acceptance 59%/26% per position); +MTP-2 7,380 / - 19.8 (71%/44%, token rate 57.4%). Deployed MTP-2: +51% decode. Both spec arms pay ~55-59% of prefill. `num_speculative_tokens` beyond 2 unmeasured. - **Measured numerics law:** spec arms do NOT reproduce the no-spec greedy text (bf16 batched verification reorders reductions; near-tie argmaxes @@ -1804,9 +1724,7 @@ any repo service. `reasoningContentField: "reasoning"`, and the extraBody moved to the Qwen3.5-family recipe (0.7/0.8) with `chat_template_kwargs` (a top-level `preserve_thinking` is ignored by vLLM). -- Deployer gap found pre-deploy: the spec-draft gate demanded a single `model.safetensors` (the dflash layout) and would have rejected the - deployed MTP arm (4 shards behind an index); fixed to accept both layouts, gated by bats. `z-lab/Qwen3.5-9B-DFlash`'s "vLLM PR #40898 unmerged" card claim is FALSE - `vllm/v1/spec_decode/dflash.py` ships in 0.30 (verified in-image); the drafter stays on disk as the alternative. @@ -1845,7 +1763,6 @@ any repo service. cutover), TP=2, 131,072 ctx x 2 seqs, bf16 KV, util 0.98, NBATCH 2048, `--enable-expert-offload --expert-precise-memory`, prefix caching, vision + `glm47` tool and reasoning parsers, MTP OFF, prefetch OFF. - - Measured on the deployed engine: **decode 9.9-10.0 tok/s, prefill 918 tok/s** (warm, 2.6k prompt), KV 138,561 tokens; arena 14.51 GiB = 2,383 slots per rank over 12,096 host-backed experts (73.64 GiB pinned per rank). - **The wall — and a corrected one.** The first account of the shortfall @@ -1951,7 +1868,6 @@ any repo service. per-sequence KDA state and the MTP draft's KV, and `gpu_worker` caps KV at that reserve, making a legal `--max-model-len` unbootable. Shipped `scripts/patches/gpu_worker-kv-cap.patch` (lets an explicit operator KV - request raise the cap) — NOT needed by the deployed shape. - Rollback: commit `2339fae` (qwen FP8 + MTP4); the qwen payload and the R9V state dirs are intact on disk. @@ -2131,7 +2047,6 @@ any repo service. (`20260921-frank-dragoon-assembled` 88 G, `-redownload` 30 G) and never purged. Total now staged on this host: **627 G**. - Verified after the moves: frank-dragoon and frank-zealot both `active`, - both `/health` 200, and all three live assets (Swift target, DFlash2 drafter, Gemma4 target) still present. - **frank-dragoon payload swapped to Swift-1.5 (uncensored)** (production change, Dankclaw). Frank asked whether the UD Q8 of @@ -2248,7 +2163,6 @@ any repo service. 8 s, drafter 2 s, session 3 s) against a 20-minute `LLM_WAIT` budget; RSS 30.6 GiB; a live short-prompt request returned `Server-Timing: ttft;dur=549.8, inter_token;dur=24.2` (~41 tok/s). - Verified live: `/health` 200, `/ready` 200, `/v1/models` → `frank-dragoon` at `context_length 262144`, a chat completion carrying `reasoning_content`, a `tool_calls` response with `finish_reason: tool_calls`, and SSE streaming. @@ -2315,9 +2229,6 @@ any repo service. - **LANDMINE, not changed here:** `/home/dank/models` was cleared by Frank mid-session to free space. frank-archon is still answering `:7770` from mmapped-but-unlinked inodes (`/models` is empty inside the running - container) — **the endpoint cannot be restarted until the 175 GiB payload - is re-fetched.** A reboot or a container restart takes frank-archon down. -- Deploy-state reconciliation (Frank's call: no restart) — the autoresearch trio (`f187165` PLE 1800, `41e9189` prefix caching, `ba2100e` docs) was found already MERGED to main + pushed AND already live on frank-archon: the running `frank-archon-r9v` container (booted 12:13 from the `ba2100e` checkout) carries @@ -2330,9 +2241,7 @@ any repo service. failure `11-admin` is the FrankBook-Pro Mac mini (`192.168.0.112:8989`) offline (100 % ICMP loss + TCP timeout), the documented asleep-host condition, not a regression. -- `scripts/frank-archon.env`, runbook — **prefix caching DEPLOYED on frank-archon** (Frank's go-ahead; commit `41e9189`): E8-qualified, zero - VRAM cost, 9x/63x reuse TTFT. Verified on :7770 (argv flag, health 200, completion "PREFIX-CACHE-LIVE"). Gotcha re-confirmed: the container is in dank's ROOTLESS podman store (`sudo podman` inspects an empty store and reports nothing); probe via the shim's `docker` or the unit MainPID. @@ -2464,7 +2373,6 @@ any repo service. ## 2026-08-26 - `/dankclaw/Caddyfile` — Both Plex vhosts proxy to PMS over a TLS upstream (`insecure_skip_verify`); PMS drops plain-HTTP connections from non-loopback sources (empty-body 502s). - `/dankclaw/gitea/quadlet/gitea-caddy.container` — Standalone Gitea Caddy quadlet retired; web proxy ownership moved to canonical Caddy; deploy-plan/deploy.sh/deploy-remote.sh + tests updated for the migration. -- Incident — Post-reboot: Caddy (a bare compose container — `restart: unless-stopped` does not survive shutdown's explicit stop) and Pi-hole (quadlet crash-loop, restart counter 1645) were both down. Root cause: `net.ipv4.ip_unprivileged_port_start` reset to 80 by a later-sorted sysctl drop-in AND a stale line in `/etc/sysctl.conf`, which systemd-sysctl applies AFTER all of `sysctl.d/`. - `/AGENTS.md` — New MANDATORY "Boot Persistence & Self-Healing" section. - `/dankclaw/caddy/quadlet/caddy.container`, `/dankclaw/caddy/env.example` — Caddy is now a boot-persistent quadlet (joins `dankclaw_default`, `Pull=never`, env from `/home/dank/Containers/caddy/env`); removed from compose.yaml. - `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — Repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` stages conflicting drop-ins, comments out the stale sysctl.conf line, and applies `sysctl --system` (the boot code path). From 8775e64ed06a4ccd62a45b10b458d993ee46ddad Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 19 Oct 2026 10:56:37 -0700 Subject: [PATCH 10/12] feat(scripts): tools to destroy documentation, not preserve it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Point-of-use enforcement needs hands, not advice: - `destroy-vault.py` — moves the pages on a kill list to `.AGENTTRASH/` and then deletes every reference to them from the survivors, including the whole line when the dead link was the line's only content. Stems are seeded from the trash so the pass is re-runnable; `pathlib.Path.stem` is NOT used to compare link targets, because it strips the real suffix off names like `sd.cpp`. - `axe-measurements.py` — deletes bench/throughput ledger sections and rate rows, blanks any table cell that reports a rate, and folds Eras older than the current one into a single row that says the numbers were destroyed and that rollback is a commit. Never touches a shape number or a Law/Wire/Rollback section. - `repair-damage.py` — fixes the two artifacts those passes can leave (a list marker that lost its space, a table separator left without its header). - `restore-punctuation.py` — restores a line whose only difference from HEAD is stripped trailing punctuation. Written after `repair-damage.py`'s first version matched every wrapped prose line ending in a comma and deleted 660 of them: the rule is now `… see .` only, and the incident is recorded in its docstring. - `gen-vault-index.py` — never walks into `.AGENTTRASH/`, and sweeps rows whose page cell is empty (a description left nameless by a destruction). Full suite: 451 tests pass. --- scripts/axe-measurements.py | 164 +++++++++++++++++++++++++++++++++ scripts/destroy-vault.py | 109 ++++++++++++++++++++++ scripts/gen-vault-index.py | 7 +- scripts/repair-damage.py | 56 +++++++++++ scripts/restore-punctuation.py | 66 +++++++++++++ 5 files changed, 401 insertions(+), 1 deletion(-) create mode 100755 scripts/axe-measurements.py create mode 100755 scripts/destroy-vault.py create mode 100755 scripts/repair-damage.py create mode 100755 scripts/restore-punctuation.py diff --git a/scripts/axe-measurements.py b/scripts/axe-measurements.py new file mode 100755 index 0000000..7c56330 --- /dev/null +++ b/scripts/axe-measurements.py @@ -0,0 +1,164 @@ +#!/usr/bin/env python3 +"""Destroy measurement ledgers in the surviving pages. + +Point of use (AGENTS.md → Documentation Policy): a rate/throughput figure for +anything not running and not being compared is destroyed, not filed. This pass +deletes + + 1. any section whose heading is a bench/throughput/rate/ladder/verdict ledger, + 2. any table row or standalone bullet that REPORTS a rate (`123.4 t/s`, + `tok/s`, `ms/token`, `TTFT`, acceptance %, `llama-bench`, a bench arm), + +and it never touches a shape number (ctx, slots, MiB budgets, ports) or a bullet +under a *Laws* / *Wire contract* / *Rollback* heading, where the sentence is the +payload and the number is decoration. Dry run by default; `--apply` writes. + +Usage: axe-measurements.py [--apply] [--verbose] +""" +import pathlib +import re +import sys + +VAULT = pathlib.Path("serverAppConfigs-docs") +RATE = re.compile( + r"\d[\d,.]*\s*(t/s|tok/s|tokens?/s|tok s|tps|ms/token|µs/token|GB/s|GiB/s)\b|" + r"\b(llama-bench|bench (arm|ladder|window)|decode ladder|prefill ladder)\b|" + r"\bacceptance (rate|of)\b|\bdraft_n\b|\bttft\b|\bServer-Timing\b", + re.I, +) +# A heading is a ledger only when the noun LEDS it and nothing instructional +# follows: `## Throughput` yes, `## 6. The context ladder (find the max bootable +# window by measurement)` no — that one is the method, which is point-of-use. +LEDGER_HEAD = re.compile( + r"^#{2,4}\s+(?:\d+[.) ]\s*)?(bench(es|mark|marks)?|throughput|results?|timings?|" + r"measured (figures|numbers|results|rates|rates)|rates|verdict|acceptance|ladder)" + r"\b[^|]{0,40}$", + re.I, +) +INSTRUCTIONAL = re.compile(r"how|why|law|policy|method|procedure|find|by measurement|execution|flip|checklist", re.I) +PROTECTED = re.compile( + r"^#{2,4}\s+.*(law|wire|rollback|contract|shape|eras|files|health|see also|related|" + r"boot|preflight|environment)", + re.I, +) +def blank_rate_cells(text): + """Kill throughput COLUMNS wherever they appear: a cell that reports a rate + becomes an em-dash, so the row keeps whatever else it records (the payload + name, the reason, the rollback) and loses only the dead number.""" + out = [] + for line in text.split("\n"): + st = line.strip() + if st.startswith("|") and st.endswith("|") and RATE.search(line): + cells = line.split("|") + for i, c in enumerate(cells): + if RATE.search(c): + cells[i] = " — " + line = "|".join(cells) + out.append(line) + return "\n".join(out) + + +def collapse_eras(text): + """One Eras row per era is fleet law; the NUMBERS in the old rows are not. + Keep the current era verbatim, fold everything older into one row that says + the numbers were destroyed and where the rollback actually lives (git).""" + lines = text.split("\n") + out = [] + in_eras = False + ncol = 0 + folded = 0 + for line in lines: + if line.startswith("#"): + in_eras = bool(re.match(r"^#{2,4}\s+.*eras\b", line, re.I)) + if folded: + out.append("| " + " | ".join( + [f"≤ 2026-09 — {folded} earlier eras: payloads, shapes and every measured " + "number destroyed 2026-10-19 as obsolete"] + ["—"] * max(0, ncol - 1))) + folded = 0 + out.append(line) + continue + st = line.strip() + if in_eras and st.startswith("|"): + cells = [c.strip() for c in st.strip("|").split("|")] + if set("".join(cells)) <= set(":- "): + ncol = len(cells) + out.append(line) + continue + if ncol == 0: + ncol = len(cells) + if re.search(r"2026-1[01]|current|active|running", st, re.I): + out.append(line) + else: + folded += 1 + continue + if folded and not st.startswith("|"): + out.append("| " + " | ".join( + [f"≤ 2026-09 — {folded} earlier eras: payloads, shapes and every measured number " + "destroyed 2026-10-19 as obsolete"] + ["—"] * max(0, ncol - 1))) + folded = 0 + ncol = 0 + out.append(line) + if folded: + out.append("| " + " | ".join( + [f"≤ 2026-09 — {folded} earlier eras destroyed 2026-10-19"] + ["—"] * max(0, ncol - 1))) + return "\n".join(out) + + +def main(argv): + apply = "--apply" in argv + verbose = "--verbose" in argv + total_lines = 0 + pages = 0 + for page in sorted(VAULT.rglob("*.md")): + s = str(page) + if ".AGENTTRASH" in s or ".obsidian" in s: + continue + text = page.read_text(errors="replace") + before = len(text.split("\n")) + if s.startswith("serverAppConfigs-docs/LLM/"): + text = collapse_eras(blank_rate_cells(text)) + lines = text.split("\n") + out = [] + skip = False + under_protected = False + cut = 0 + for line in lines: + if line.startswith("#"): + under_protected = bool(PROTECTED.search(line)) + skip = (bool(LEDGER_HEAD.match(line)) and not under_protected + and not INSTRUCTIONAL.search(line)) + if skip: + cut += 1 + continue + if skip: + cut += 1 + continue + stripped = line.strip() + out.append(line) + if under_protected: + # Eras rows are the rollback record, Law bullets are the payload: + # the number is decoration inside them, and the row/bullet is not + # a ledger entry. Leave the whole section alone. + continue + if stripped.startswith(("|", "-", "*")) and RATE.search(line): + # Destroy only when the line EXISTS to report a number: a table + # row, or a bullet whose verb is measurement. + if stripped.startswith("|") or re.search( + r"\b(measured|measures|bench|benched|ran at|runs at|clocked|hit|" + r"landed at|came in at|peak|peaks at|sustained|wall.clock)\b", line, re.I): + out.pop() + cut += 1 + final = re.sub(r"\n{3,}", "\n\n", "\n".join(out)) + saved = before - len(final.split("\n")) + if saved or cut: + pages += 1 + total_lines += max(saved, cut) + print(f" {s.replace('serverAppConfigs-docs/', '')}: -{max(saved, cut)}") + if apply: + page.write_text(final.rstrip("\n") + "\n") + print(f"{'APPLIED' if apply else 'dry run'}: {pages} pages, {total_lines} lines of ledger") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/scripts/destroy-vault.py b/scripts/destroy-vault.py new file mode 100755 index 0000000..310c740 --- /dev/null +++ b/scripts/destroy-vault.py @@ -0,0 +1,109 @@ +#!/usr/bin/env python3 +"""Destroy the vault's dead weight: era ledgers, retired runtimes, dead experiments. + +`AGENTS.md` was treating a measurement as a record worth keeping forever. It is not: a +throughput figure for a payload that no longer runs, on a runtime that no longer exists, +is a liability — the next agent spends context on a number that AI progress has already +obsolete. This pass moves those pages to `.AGENTTRASH/` (repo law: never `rm`) and then +deletes every reference to them from the surviving pages, including the sentence that +carried the link when the link was the sentence's only content. + +Usage: destroy-vault.py +List file: one vault-relative path per line; a directory path destroys the whole tree. +""" +import pathlib +import re +import shutil +import sys + +VAULT = pathlib.Path("serverAppConfigs-docs") +TRASH = VAULT / ".AGENTTRASH" / "20261019-destruction" + + +def targets(list_file): + out = [] + for line in pathlib.Path(list_file).read_text().splitlines(): + line = line.strip() + if not line or line.startswith("#"): + continue + p = VAULT / line + if p.is_dir(): + out += [f for f in sorted(p.rglob("*.md"))] + elif p.is_file(): + out.append(p) + else: + print(f" SKIP (missing): {line}") + return out + + +def main(argv): + if len(argv) < 2: + print(__doc__) + return 2 + doomed = targets(argv[1]) + # Stems come from the trash too, so re-running the pass (or running it after a + # partial destruction) still knows which names are dead and keeps scrubbing. + stems = {p.stem for p in doomed} + if TRASH.is_dir(): + stems |= {p.stem for p in TRASH.rglob("*.md")} + moved = 0 + for p in doomed: + dest = TRASH / p.relative_to(VAULT) + dest.parent.mkdir(parents=True, exist_ok=True) + shutil.move(str(p), str(dest)) + moved += 1 + print(f"destroyed {moved} pages ({sum(len(p.read_text(errors='replace').splitlines()) for p in TRASH.rglob('*.md'))} lines staged in trash)") + + # Repoint-or-delete: a link to a destroyed page is itself dead information. + link_wl = re.compile(r"\[\[([^\]|#]+)(?:#[^\]|]*)?(?:\|([^\]]*))?\]\]") + link_md = re.compile(r"\[[^\]]*\]\(([^)#]*\.md)(?:#[^)]*)?\)") + pages_cleaned = links_dropped = lines_dropped = 0 + for page in sorted(VAULT.rglob("*.md")): + s = str(page) + if ".AGENTTRASH" in s or ".obsidian" in s: + continue + text = page.read_text(errors="replace") + lines = text.split("\n") + out = [] + hits = 0 + + def dead(target): + # NOT pathlib.Path(...).stem: that strips a real suffix off names like + # `Lemonade and sd.cpp`, so a link to a destroyed page survives and the + # vault then fails its own link check. + t = target.strip() + if t.endswith(".md"): + t = t[:-3] + return t in stems or pathlib.PurePosixPath(t).name in stems + + for line in lines: + found = [t for t in link_wl.findall(line) if dead(t[0])] + for m in link_md.finditer(line): + if dead(m.group(1)): + found.append(m.group(1)) + if not found: + out.append(line) + continue + hits += len(found) + stripped = link_wl.sub( + lambda m: m.group(2) if (m.group(2) and not dead(m.group(1))) else "", + line) + stripped = link_md.sub( + lambda m: "" if dead(m.group(1)) else m.group(0), stripped) + # What is left of the line after the dead links are gone? + residue = re.sub(r"[\s\-*•·|:;,.\u2014\u2013]+", "", stripped) + if residue: + out.append(re.sub(r"\s+$", "", re.sub(r"^(\s*[-*]\s*)[\s:;,.\u2014-]+", r"\1", stripped))) + else: + lines_dropped += 1 + if hits: + page.write_text("\n".join(out)) + pages_cleaned += 1 + links_dropped += hits + print(f"cleaned {pages_cleaned} surviving pages: {links_dropped} dead links dropped, " + f"{lines_dropped} lines left empty and deleted") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/scripts/gen-vault-index.py b/scripts/gen-vault-index.py index ff53505..ff7c7de 100755 --- a/scripts/gen-vault-index.py +++ b/scripts/gen-vault-index.py @@ -107,6 +107,11 @@ def sync_subindex(index: pathlib.Path, vault: pathlib.Path): lines = index.read_text().split("\n") out, dropped, refreshed = [], 0, 0 for line in lines: + # A row whose first cell is empty is a description with no page: the link + # was deleted when the page was destroyed, so the row is a dead promise. + if re.match(r"^\|\s*\|", line.strip()): + dropped += 1 + continue for pat, wrap in ((ROW, "| [[{}]] |{} {} |"), (BULLET, "- [[{}]] — {}")): m = pat.match(line) if not m: @@ -191,7 +196,7 @@ def main(argv): index.write_text("\n".join(out) + "\n") touched = 0 for sub in sorted(vault.rglob("index.md")): - if sub == index: + if sub == index or ".AGENTTRASH" in str(sub): continue dropped, refreshed, added = sync_subindex(sub, vault) if dropped or refreshed or added: diff --git a/scripts/repair-damage.py b/scripts/repair-damage.py new file mode 100755 index 0000000..b86b109 --- /dev/null +++ b/scripts/repair-damage.py @@ -0,0 +1,56 @@ +#!/usr/bin/env python3 +"""Repair the two artifacts the destruction passes introduced. + +1. The link stripper could eat the space after a `- ` marker (`-`X``) and leave a + dangling `see .` where a wikilink used to be. +2. The Eras collapse could swallow a table's HEADER row (it looked like just + another non-current row), leaving a separator with no header above it. + +Usage: repair-damage.py [--apply] +""" +import pathlib +import re +import sys + +VAULT = pathlib.Path("serverAppConfigs-docs") +ERAS_HEADER = "| Window | Payload | Runtime | Shape | Replaced because | Rollback |" + + +def main(argv): + apply = "--apply" in argv + marker = dangling = header = 0 + for p in sorted(VAULT.rglob("*.md")): + s = str(p) + if ".AGENTTRASH" in s or ".obsidian" in s: + continue + lines = p.read_text(errors="replace").split("\n") + out = [] + # Only a pointer whose target was deleted reads as `… see .` — never touch + # a line merely because it wraps on a comma. (An earlier version of this + # rule stripped the trailing comma off 660 wrapped prose lines, which + # `scripts/restore-punctuation.py` then had to undo from HEAD.) + for line in lines: + new = line + if re.match(r"^(\s*)[-*](?![-* ])[$`[(*_]", new): + new = re.sub(r"^(\s*)[-*]", r"\1- ", new) + marker += 1 + if re.search(r"\s(?:see|also|details in|via|per)\s*\.?\s*$", new): + new = re.sub(r"\s(?:see|also|details in|via|per)\s*\.?\s*$", "", new) + dangling += 1 + out.append(new) + # A separator row whose line above is not a header gets its header back. + fixed = [] + for i, line in enumerate(out): + if re.match(r"^\|-+", line.strip()) and (not fixed or not fixed[-1].strip().startswith("|")): + fixed.append(ERAS_HEADER) + header += 1 + fixed.append(line) + if apply and "".join(fixed) != "".join(lines): + p.write_text("\n".join(fixed)) + print(f"{'APPLIED' if apply else 'dry run'}: marker spacing {marker}, " + f"dangling pointers {dangling}, restored table headers {header}") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/scripts/restore-punctuation.py b/scripts/restore-punctuation.py new file mode 100755 index 0000000..1ecade0 --- /dev/null +++ b/scripts/restore-punctuation.py @@ -0,0 +1,66 @@ +#!/usr/bin/env python3 +"""Undo trailing-punctuation damage by restoring lines that match HEAD exactly +except for a stripped trailing `,` / `;` / removed trailing word. + +The repair pass over-reached: its "dangling pointer" cleanup also matched every +wrapped prose line ending in a comma, and deleted the comma. A line whose text +is otherwise byte-identical to its HEAD version, differing only by trailing +punctuation, is not an edit — it is damage, so HEAD's line goes back. + +Usage: restore-punctuation.py [--apply] +""" +import pathlib +import subprocess +import sys + +REPO = pathlib.Path(".") +VAULT = REPO / "serverAppConfigs-docs" + + +def head_text(rel): + r = subprocess.run(["git", "show", f"HEAD:{rel}"], capture_output=True, text=True) + return r.stdout if r.returncode == 0 else None + + +def main(argv): + apply = "--apply" in argv + restored = 0 + pages = 0 + for p in sorted(VAULT.rglob("*.md")): + s = str(p) + if ".AGENTTRASH" in s or ".obsidian" in s: + continue + rel = s + head = head_text(rel) + if head is None: + continue + cur = p.read_text(errors="replace").split("\n") + # Key on (length, prefix) of the HEAD line minus trailing punctuation so a + # damaged line can find its original. + index = {} + for hl in head.split("\n"): + h = hl.rstrip() + if h.endswith((",", ";")) or True: + key = (len(h.rstrip(",;. ")), h.rstrip(",;. ")[:60]) + index.setdefault(key, h) + touched = 0 + for i, cl in enumerate(cur): + c = cl.rstrip() + if not c or c.endswith((".", ":", "|", "`", ")", "," , ";")): + continue + key = (len(c), c[:60]) + hl = index.get(key) + if hl and hl != c and hl.startswith(c): + cur[i] = hl + touched += 1 + if touched: + pages += 1 + restored += touched + if apply: + p.write_text("\n".join(cur)) + print(f"{'APPLIED' if apply else 'dry run'}: {restored} lines restored across {pages} pages") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) From 327e78b950d907a1575b13ebc38c903a39655c5a Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 19 Oct 2026 10:56:37 -0700 Subject: [PATCH 11/12] =?UTF-8?q?docs(vault):=20destroy=20the=20dead=20wei?= =?UTF-8?q?ght=20=E2=80=94=2053=20pages=20and=20the=20change=20diary?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The vault was 155 pages / 31,210 lines, most of it measurement ledgers for payloads that no longer run. A throughput figure for a runtime nobody operates is not a record: AI moves faster than the vault, so the number is wrong before anyone reads it, and a wrong number in a runbook gets believed. Per the new purpose test in `AGENTS.md` → *Documentation Policy*, it is destroyed. Gone (`.AGENTTRASH/20261019-destruction/`, KILL-LIST.txt is the manifest): - the `frank-medium` endpoint and its five era pages, `frank-npu`, FastFlowLM, the ikllama/kingjones/dealign/frank-redo eras, the ling/swift15/freetoken/ubatch campaign pages, qwen3.8 spec-matrix and strix-optimizations, the two "Running Qwen3.6 …" pages; - the whole `Pipeline Parallelism/` folder (no live script passes a pipeline flag) and the multi-device/x4-pair/hybrid/offload experiment pages; - every vLLM page — no vLLM endpoint exists on this fleet; - `LLM/deployment.md` (`Architecture/Deployment Flow.md` is the test-referenced owner) and the external surveys (SGLang, Lemonade, llama.cpp forks/diff, self-hosted app census, the TTS survey) that were obsolete by construction; - `log.md`, 2,380 lines. Git holds the chronology; a causal law belongs to the page that owns the thing it governs, never to a diary. Survivors: 101 pages, 17,542 lines, 0 OKF violations, 0 unresolved links. Eras tables keep the window, payload, runtime, reason and rollback pointer; the rate columns are blanked. The live shapes stay verbatim — those numbers are what the deployer and `models.yml` consume. --- .../Agent/Agent Docs and Edits.md | 6 +- serverAppConfigs-docs/Agent/OMP Harness.md | 22 +- serverAppConfigs-docs/Agent/index.md | 2 +- .../Container Network Namespaces.md | 4 +- .../Architecture/Deploy State Triage.md | 10 +- .../Architecture/Deployment Flow.md | 3 +- .../Architecture/E2E Suite.md | 2 +- .../Architecture/Network Architecture.md | 12 +- .../Architecture/Quadlet Lifecycle.md | 10 +- .../Architecture/Testing Policy.md | 1 - .../Dankclaw/ComfyUI Qwen-Image Workflows.md | 12 +- serverAppConfigs-docs/Dankclaw/ComfyUI.md | 6 +- .../Dankclaw/Dankclaw Stack.md | 9 +- .../Dankclaw/Frank TTS Engine.md | 4 +- serverAppConfigs-docs/Dankclaw/Frank TTS.md | 10 +- serverAppConfigs-docs/Dankclaw/Gitea.md | 2 +- .../Dankclaw/Hermes-Dashboard.md | 2 +- .../Dankclaw/Hermes-WebUI.md | 8 +- serverAppConfigs-docs/Dankclaw/Hermes.md | 14 +- serverAppConfigs-docs/Dankclaw/Marreta.md | 20 +- serverAppConfigs-docs/Dankclaw/OpenWebUI.md | 23 +- serverAppConfigs-docs/Dankclaw/Postgres.md | 2 +- .../Dankclaw/SearXNG Providers.md | 12 +- serverAppConfigs-docs/Dankclaw/SearXNG.md | 33 +- .../Dankclaw/Swap Configuration.md | 4 +- serverAppConfigs-docs/Dankclaw/Voice Packs.md | 12 +- serverAppConfigs-docs/Dankserver/Calibre.md | 2 +- serverAppConfigs-docs/Dankserver/Copyparty.md | 6 +- .../Dankserver/Dankserver Stack.md | 2 +- serverAppConfigs-docs/Dankserver/Jellyfin.md | 6 +- serverAppConfigs-docs/Dankserver/Plex.md | 2 +- serverAppConfigs-docs/Dankserver/RomM.md | 4 +- .../Desktop/Noctalia and Niri.md | 22 +- .../FrankBook-Pro/ARR-Stack.md | 4 +- .../FrankBook-Pro/prowlarr.md | 6 +- .../FrankBook-Pro/qbittorrent.md | 4 +- .../Infrastructure/Anubis.md | 6 +- .../Authentik Outpost Reference.md | 18 +- .../Infrastructure/Authentik.md | 50 +- .../Infrastructure/FrankBook-Pro.md | 10 +- .../Infrastructure/Grafana.md | 6 +- .../Infrastructure/Homepage Status Probes.md | 4 +- .../Infrastructure/Homepage.md | 18 +- .../Infrastructure/LAN-DNS.md | 4 +- serverAppConfigs-docs/Infrastructure/Loki.md | 2 +- .../Observability Provisioning.md | 16 +- .../Infrastructure/Observability.md | 14 +- .../LLM/DFlash DSpark Speculative Decoding.md | 22 +- .../LLM/Dankclaw Improvements.md | 16 +- .../LLM/Dankclaw System Specs.md | 10 +- .../LLM/Dankserver System Specs.md | 17 +- .../LLM/Dual-R9700-Cooperation.md | 327 --- .../LLM/FastFlowLM NPU Inference.md | 448 ---- serverAppConfigs-docs/LLM/FreeToken Builds.md | 97 - .../LLM/GGUF Compatibility.md | 14 +- .../LLM/Heterogeneous Split Triage.md | 14 +- .../Hybrid CPU GPU Processing Dankserver.md | 225 -- .../LLM/LLM Bench Windows and Measurement.md | 271 -- .../LLM/LLM Performance Diagnosis.md | 16 +- .../LLM/Llama Cpp Multi-Device Splits.md | 287 -- .../LLM/Llama Cpp Serving Sizing.md | 15 +- .../LLM/Model Swap and Rollback.md | 18 +- .../LLM/PCIe Link and Atomics.md | 42 +- .../Ethernet Bandwidth Math.md | 454 ---- .../Expected Performance.md | 97 - .../Qwen3.8 PP Deployment.md | 313 --- .../Pipeline Parallelism/Sample Script Set.md | 261 -- .../LLM/Pipeline Parallelism/index.md | 71 - .../LLM/Quantization Production.md | 24 +- .../Quantization by Hardware Architecture.md | 5 +- .../LLM/RAM CPU Offloading in VLLM.md | 206 -- .../LLM/RDNA4 Container Serving.md | 258 -- .../RDNA4 MoE Expert Offload VRAM Budget.md | 7 +- .../Running Qwen3.6 27B FP8 on Dankclaw.md | 246 -- ...nning Qwen3.6 35B A3B FP8 on Dankserver.md | 210 -- .../Shared Memory Architecture Dankclaw.md | 13 +- .../LLM/Tcclaviger Fork Ops.md | 202 -- serverAppConfigs-docs/LLM/VLLM Index.md | 104 - serverAppConfigs-docs/LLM/VLLM Overview.md | 287 -- .../LLM/dankclaw-iommu-translation-off.md | 12 +- .../dankserver-frontier-moe-feasibility.md | 227 -- .../LLM/dankserver-vram-growth.md | 144 - .../LLM/dealign-ornith-crack.md | 127 - serverAppConfigs-docs/LLM/deployment.md | 178 -- serverAppConfigs-docs/LLM/frank-archon.md | 65 +- serverAppConfigs-docs/LLM/frank-dragoon.md | 47 +- .../LLM/frank-medium-batch-ubatch.md | 227 -- .../LLM/frank-medium-llamacpp-dflash2.md | 734 ----- .../LLM/frank-medium-model-swap.md | 189 -- .../LLM/frank-medium-radiance-mxfp4.md | 638 ----- .../LLM/frank-medium-rocmfpx.md | 221 -- serverAppConfigs-docs/LLM/frank-medium.md | 65 - serverAppConfigs-docs/LLM/frank-npu.md | 165 -- .../LLM/frank-redo-2026-08-17.md | 227 -- serverAppConfigs-docs/LLM/frank-zealot.md | 40 +- .../LLM/freetoken-benchmark-sweep.md | 94 - .../LLM/ikllama-ornith-dankserver.md | 263 -- serverAppConfigs-docs/LLM/index.md | 47 +- .../LLM/kingjones-q8agent-rocmfpx.md | 243 -- .../LLM/ling-3.0-flash-trial.md | 123 - .../LLM/qwen3.8-spec-matrix.md | 141 - .../LLM/qwen3.8-strix-optimizations.md | 232 -- .../LLM/swift15-campaign-worklog.md | 82 - .../LLM/ubatch-benchmark-large.md | 56 - ...Boot Forensics and Config Interrogation.md | 208 -- .../LLM/vLLM Porting and Compile Triage.md | 330 --- .../LLM/vllm_research_findings.md | 236 -- .../LLM/x4-pair-experiment.md | 409 --- .../Operations/Container Host Reachability.md | 2 +- .../Operations/Fleet Audit and Posture.md | 12 +- .../Operations/Fleet Ops Playbook.md | 9 +- .../Operations/Payload and Disk Reclaim.md | 4 +- .../Operations/Remote Job Runner.md | 18 +- .../Operations/SSH and Remote Access.md | 4 +- serverAppConfigs-docs/Operations/index.md | 2 +- .../AI Job Search and Resume Tools.md | 12 +- .../Research/ComfyUI ROCm and Vulkan.md | 4 +- .../Research/Lemonade and sd.cpp.md | 57 - .../Research/Llama Cpp Code Diff.md | 222 -- .../Research/Llama Cpp Runtimes.md | 141 - .../Research/Local TTS and Voice Cloning.md | 467 ---- .../Most Popular Self-Hosted AI Apps.md | 218 -- serverAppConfigs-docs/Research/SGLang.md | 154 -- serverAppConfigs-docs/Research/index.md | 13 +- serverAppConfigs-docs/index.md | 63 - serverAppConfigs-docs/log.md | 2380 ----------------- 126 files changed, 443 insertions(+), 14157 deletions(-) delete mode 100644 serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md delete mode 100644 serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md delete mode 100644 serverAppConfigs-docs/LLM/FreeToken Builds.md delete mode 100644 serverAppConfigs-docs/LLM/Hybrid CPU GPU Processing Dankserver.md delete mode 100644 serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md delete mode 100644 serverAppConfigs-docs/LLM/Llama Cpp Multi-Device Splits.md delete mode 100644 serverAppConfigs-docs/LLM/Pipeline Parallelism/Ethernet Bandwidth Math.md delete mode 100644 serverAppConfigs-docs/LLM/Pipeline Parallelism/Expected Performance.md delete mode 100644 serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.8 PP Deployment.md delete mode 100644 serverAppConfigs-docs/LLM/Pipeline Parallelism/Sample Script Set.md delete mode 100644 serverAppConfigs-docs/LLM/Pipeline Parallelism/index.md delete mode 100644 serverAppConfigs-docs/LLM/RAM CPU Offloading in VLLM.md delete mode 100644 serverAppConfigs-docs/LLM/RDNA4 Container Serving.md delete mode 100644 serverAppConfigs-docs/LLM/Running Qwen3.6 27B FP8 on Dankclaw.md delete mode 100644 serverAppConfigs-docs/LLM/Running Qwen3.6 35B A3B FP8 on Dankserver.md delete mode 100644 serverAppConfigs-docs/LLM/Tcclaviger Fork Ops.md delete mode 100644 serverAppConfigs-docs/LLM/VLLM Index.md delete mode 100644 serverAppConfigs-docs/LLM/VLLM Overview.md delete mode 100644 serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md delete mode 100644 serverAppConfigs-docs/LLM/dankserver-vram-growth.md delete mode 100644 serverAppConfigs-docs/LLM/dealign-ornith-crack.md delete mode 100644 serverAppConfigs-docs/LLM/deployment.md delete mode 100644 serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md delete mode 100644 serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md delete mode 100644 serverAppConfigs-docs/LLM/frank-medium-model-swap.md delete mode 100644 serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md delete mode 100644 serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md delete mode 100644 serverAppConfigs-docs/LLM/frank-medium.md delete mode 100644 serverAppConfigs-docs/LLM/frank-npu.md delete mode 100644 serverAppConfigs-docs/LLM/frank-redo-2026-08-17.md delete mode 100644 serverAppConfigs-docs/LLM/freetoken-benchmark-sweep.md delete mode 100644 serverAppConfigs-docs/LLM/ikllama-ornith-dankserver.md delete mode 100644 serverAppConfigs-docs/LLM/kingjones-q8agent-rocmfpx.md delete mode 100644 serverAppConfigs-docs/LLM/ling-3.0-flash-trial.md delete mode 100644 serverAppConfigs-docs/LLM/qwen3.8-spec-matrix.md delete mode 100644 serverAppConfigs-docs/LLM/qwen3.8-strix-optimizations.md delete mode 100644 serverAppConfigs-docs/LLM/swift15-campaign-worklog.md delete mode 100644 serverAppConfigs-docs/LLM/ubatch-benchmark-large.md delete mode 100644 serverAppConfigs-docs/LLM/vLLM Boot Forensics and Config Interrogation.md delete mode 100644 serverAppConfigs-docs/LLM/vLLM Porting and Compile Triage.md delete mode 100644 serverAppConfigs-docs/LLM/vllm_research_findings.md delete mode 100644 serverAppConfigs-docs/LLM/x4-pair-experiment.md delete mode 100644 serverAppConfigs-docs/Research/Lemonade and sd.cpp.md delete mode 100644 serverAppConfigs-docs/Research/Llama Cpp Code Diff.md delete mode 100644 serverAppConfigs-docs/Research/Llama Cpp Runtimes.md delete mode 100644 serverAppConfigs-docs/Research/Local TTS and Voice Cloning.md delete mode 100644 serverAppConfigs-docs/Research/Most Popular Self-Hosted AI Apps.md delete mode 100644 serverAppConfigs-docs/Research/SGLang.md delete mode 100644 serverAppConfigs-docs/log.md diff --git a/serverAppConfigs-docs/Agent/Agent Docs and Edits.md b/serverAppConfigs-docs/Agent/Agent Docs and Edits.md index 0104539..8f08456 100644 --- a/serverAppConfigs-docs/Agent/Agent Docs and Edits.md +++ b/serverAppConfigs-docs/Agent/Agent Docs and Edits.md @@ -21,7 +21,7 @@ Damage catalogue: no longer frontmatter (title/tags/description become body text). 2. A rewrite **appended** instead of merged → duplicate headings and a stray second `---` mid-file. -3. Literal model artifacts in the body: `<|channel>}...`, +3. Literal model artifacts in the body: `<|channel>}...` `+- `, `- - `. 4. Wrapped bullets re-emitted with a `- ` prefix on every physical line. 5. Line-range replacement that dropped a sentence tail → dangling fragment, @@ -46,7 +46,7 @@ Repair order (matters): 2. If it merely duplicated: **salvage unique content first** — cut the one-of-a-kind section, paste it into the copy you keep, THEN delete the duplicate ranges. Never delete first. -3. Re-add missing sentence tails (insert before the orphaned fragment), +3. Re-add missing sentence tails (insert before the orphaned fragment) keeping list continuation lines indented. 4. Re-run all checks above. @@ -57,7 +57,7 @@ safe, but its snapshot tag changes on every edit — re-read before each subsequent edit. `.AGENTTRASH/` is gitignored (`.gitignore:25`), so NEW trash stays local — but do not read that as "the repo has none of it": on 2026-10-18 `git ls-files | grep -c AGENTTRASH` returns **37** tracked paths -(`.AGENTTRASH/`, `scripts/.AGENTTRASH/`, `dankserver/.AGENTTRASH/`, +(`.AGENTTRASH/`, `scripts/.AGENTTRASH/`, `dankserver/.AGENTTRASH/` `tests/.AGENTTRASH/`), because gitignore never untracks what was committed before the rule. Trash is a temporary hold, not an archive. diff --git a/serverAppConfigs-docs/Agent/OMP Harness.md b/serverAppConfigs-docs/Agent/OMP Harness.md index 4e44a74..239e01b 100644 --- a/serverAppConfigs-docs/Agent/OMP Harness.md +++ b/serverAppConfigs-docs/Agent/OMP Harness.md @@ -16,7 +16,7 @@ this file is repo-managed; edits are harness-side only. **Model ids are the StarCraft unit names (official 2026-10-08, on the wire since 2026-10-09):** `frank-archon` (Dankserver), `frank-dragoon` and `frank-zealot` (Dankclaw) — engine, port and card behind each id live on -[[frank-archon]], [[frank-dragoon]] and [[frank-zealot]] — formerly `frank-archon`, `frank-dragoon`, +[[frank-archon]], [[frank-dragoon]] and [[frank-zealot]] — formerly `frank-archon`, `frank-dragoon` `frank-zealot`. The SERVICES keep the old names (units, scripts, env files, runbook pages); pages written in the old era name the service, the harness names the model. The wire now matches: each endpoint's @@ -26,7 +26,7 @@ picker/grants, the homepage docs and the raw API all use one id per model note on the endpoint echoing the requested name). Verified 2026-10-18: `/v1/models` on 192.168.0.235:7770, 192.168.0.143:7770 and -192.168.0.143:7771 each answer with the unit name (`frank-archon`, +192.168.0.143:7771 each answer with the unit name (`frank-archon` `frank-dragoon`, `frank-zealot`). ## Compaction chain arithmetic (verify before ANY window change) @@ -40,11 +40,11 @@ breaks compaction. Where the contract lives: - `~/.omp/agent/config.yml` → `compaction.remoteEndpoint` (target URL: - `http://192.168.0.143:7771/v1/chat/completions`, `config.yml:76`), - `methodOrder` (`shake → snapcompact → soft → handoff → remote`, + `http://192.168.0.143:7771/v1/chat/completions`, `config.yml:76`) + `methodOrder` (`shake → snapcompact → soft → handoff → remote` `config.yml:77-82`), `thresholdPercent` (**95**, `config.yml:84`), idle settings (`idleEnabled`, `idleThresholdTokens: 100000`) -- `~/.omp/agent/models.yml` → per-model `contextWindow`, `maxTokens`, +- `~/.omp/agent/models.yml` → per-model `contextWindow`, `maxTokens` `compactionModel` references - Implementation: omp plugin `~/.omp/plugins/node_modules/@earendil-works/pi-coding-agent/dist/core/compaction/compaction.js` @@ -104,16 +104,16 @@ input. Boot-shape OOM laddering is a different problem ## Per-model sampler config (`compat.extraBody`) `models.yml` has **no per-model sampler keys**. The global knobs are -`config.yml`'s `temperature`, `topP`, `topK`, `minP`, `presencePenalty`, +`config.yml`'s `temperature`, `topP`, `topK`, `minP`, `presencePenalty` `repetitionPenalty`, and a knob reaches the wire only when it is set — the -bundle guards each one with `!== void 0` (e.g. `minP!==void 0)e.min_p=t.minP`), +bundle guards each one with `!== void 0` (e.g. `minP!==void 0)e.min_p=t.minP`) and this workstation's `config.yml` sets **none** of them. The 2026-10-05 note that the unset default is `-1`/"not sent" is retired: there is no `-1` sentinel, an unset key is simply absent. The per-model hook is `compat.extraBody`: **Why it works:** `applyOpenAIExtraBody` runs **last** in the -openai-completions builder — after `applyChatCompletionsCompatPolicy` — via +openai-completions builder — after `applyChatCompletionsCompatPolicy` — `Object.assign(params, extraBody)`. So extraBody overrides global settings and compat-policy output. @@ -191,7 +191,7 @@ assistant turn carrying a long `reasoning_content`, then compare `usage.prompt_tokens` with the flag true vs false. A large delta means the field changed template rendering; zero delta means it was ignored. -Cleanup: trash the probe profile and echo server (`profiles/.AGENTTRASH`), +Cleanup: trash the probe profile and echo server (`profiles/.AGENTTRASH`) never `rm`. ## Static-preamble audit ("the context feels bloated") @@ -205,7 +205,7 @@ What loads statically every session (measure each, don't guess): ```bash python3 - <<'EOF' import glob, re, os - for root in [os.path.expanduser("~/.omp/agent/managed-skills"), + for root in [os.path.expanduser("~/.omp/agent/managed-skills") os.path.expanduser("~/.omp/agent/skills"), ".omp/skills"]: n=tot=0 for sk in glob.glob(root+"/*/SKILL.md"): @@ -305,4 +305,4 @@ standalone; fold go/no-go arithmetic skills into their probe skill - [[Agent Docs and Edits]] — editing vault docs and hashline targets safely - [[Command Center Dotfiles]] — the `~/.omp` repo deploy this config rides - [[Llama Cpp Serving Sizing]] — the engine-side sizing the chain must fit -- [[LLM Bench Windows and Measurement]] — measuring the endpoints this config routes +- measuring the endpoints this config routes diff --git a/serverAppConfigs-docs/Agent/index.md b/serverAppConfigs-docs/Agent/index.md index 26c4697..c2424cd 100644 --- a/serverAppConfigs-docs/Agent/index.md +++ b/serverAppConfigs-docs/Agent/index.md @@ -9,4 +9,4 @@ docs get edited safely, and how the agent's home is deployed. ## Related -- [LLM](../LLM/index.md) — the fleet these agents operate +- the fleet these agents operate diff --git a/serverAppConfigs-docs/Architecture/Container Network Namespaces.md b/serverAppConfigs-docs/Architecture/Container Network Namespaces.md index e07af61..e69fa7a 100644 --- a/serverAppConfigs-docs/Architecture/Container Network Namespaces.md +++ b/serverAppConfigs-docs/Architecture/Container Network Namespaces.md @@ -47,7 +47,7 @@ network namespace**: `Network=` values are read from the live Quadlet sources in `/home/dank/.config/containers/systemd/*.container`; runtime network names on the -host at the same date were `podman`, `dankclaw_default`, `systemd-gitea`, +host at the same date were `podman`, `dankclaw_default`, `systemd-gitea` `systemd-searxng`, `systemd-open-webui`. Two consequences that are easy to miss: @@ -83,7 +83,7 @@ podman exec cat /proc/net/route # default route → gateway IP podman exec sh -c 'timeout 2 bash -c "exec 3<>/dev/tcp/10.89.1.1/8080"; echo rc=$?' ``` -Verified outcomes of that recipe on 2026-09-02: from inside `hermes-dashboard`, +Verified outcomes of that recipe on 2026-09-02: from inside `hermes-dashboard` `10.89.1.1:3000` (Gitea, published), `:8081` (SearXNG via Gluetun's netns, published) and `:9119` (its own published port) connect; `:8080` and `:8787` are **refused instantly** even though both listeners are bound to `0.0.0.0` in diff --git a/serverAppConfigs-docs/Architecture/Deploy State Triage.md b/serverAppConfigs-docs/Architecture/Deploy State Triage.md index 6ab442a..3618b6c 100644 --- a/serverAppConfigs-docs/Architecture/Deploy State Triage.md +++ b/serverAppConfigs-docs/Architecture/Deploy State Triage.md @@ -36,7 +36,7 @@ possibly zero behavioral change. 2. Probe the target (ssh as `dank`, passwordless sudo present): - `cat ~/.local/state/serverAppConfigs/deploy-state/` - `git -C ~/serverAppConfigs rev-parse HEAD` - - `systemctl is-active/is-enabled `; + - `systemctl is-active/is-enabled ` `curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:/health` - the on-host env file already carries the new keys (it landed via the ff-pull). @@ -119,8 +119,8 @@ each, kill stale ones by explicit PID. A killed deploy leaves behind: - the unit **stopped/failed** (`Restart=always` does NOT fire after an - explicit stop — start it yourself); -- the state marker **pending** (pre-deploy or in-flight revision); + explicit stop — start it yourself) +- the state marker **pending** (pre-deploy or in-flight revision) - the e2e gate un-run (the deployer runs it last). Reconcile the marker ONLY after the service itself is verified healthy — a @@ -142,11 +142,11 @@ Boot failures must be read from the journal, not `podman logs`: serve scripts numbers — which looks exactly like your change having no effect. Key on container identity (`podman ps -a --format '{{.CreatedAt}}'`) and read the cause from `journalctl --since "N min ago" | grep -a "\["`. -See [[vLLM Boot Forensics and Config Interrogation]] for the full capture +See for the full capture procedure. ## Related - [[Deployment Flow]] — the deploy pipeline itself - [[Quadlet Lifecycle]] — unit ownership and drift diagnosis -- [[vLLM Boot Forensics and Config Interrogation]] — journal capture +- journal capture diff --git a/serverAppConfigs-docs/Architecture/Deployment Flow.md b/serverAppConfigs-docs/Architecture/Deployment Flow.md index 44195be..513e654 100644 --- a/serverAppConfigs-docs/Architecture/Deployment Flow.md +++ b/serverAppConfigs-docs/Architecture/Deployment Flow.md @@ -68,7 +68,7 @@ service on the other host. | Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver//quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/` to `/home/dank/Containers/` (mapped-uid roots move without chown). Steady state is restart-only. | The shared chat template maps to the inference service on the selected target. -The `frank-npu` endpoint is retired ([[frank-npu]]) and has no deploy handler; +The `frank-npu` endpoint is retired () and has no deploy handler an unrelated large-model deployment never touches it. ## Caddy safety @@ -125,6 +125,5 @@ routing. See [[Testing Policy]]. ## See Also - [[Testing Policy]] -- [LLM deployment](../LLM/deployment.md) - [[Caddy]] - [[SearXNG]] diff --git a/serverAppConfigs-docs/Architecture/E2E Suite.md b/serverAppConfigs-docs/Architecture/E2E Suite.md index a03d3f0..7b47684 100644 --- a/serverAppConfigs-docs/Architecture/E2E Suite.md +++ b/serverAppConfigs-docs/Architecture/E2E Suite.md @@ -49,7 +49,7 @@ user sees it; the `bats` seams assert config shape, not behavior. *arr/qBittorrent links terminate on FrankBook-Pro, which sleeps when idle, and a sleeping Mac mini must never fail an unrelated deploy. The project joins only when `E2E_INCLUDE_FRANKBOOK=1` (set by `scripts/deploy.sh` when a frankbook - trigger path changed: `tests/e2e/specs/19-frankbook.spec.ts`, `frankbook-pro/**`, + trigger path changed: `tests/e2e/specs/19-frankbook.spec.ts`, `frankbook-pro/**` `shared/homepage/services.json`, or the `dankclaw/Caddyfile` serving the arr vhosts) or when an explicit `--project` argument names it. An `11-admin` edit that touches the arr links must therefore be paired with a `19-frankbook` run diff --git a/serverAppConfigs-docs/Architecture/Network Architecture.md b/serverAppConfigs-docs/Architecture/Network Architecture.md index 13e424d..34eb1f3 100644 --- a/serverAppConfigs-docs/Architecture/Network Architecture.md +++ b/serverAppConfigs-docs/Architecture/Network Architecture.md @@ -15,7 +15,7 @@ One Caddy fronts both hosts. It runs as a container on Dankclaw (192.168.0.143) As of 2026-08-28 the front door is a two-tier chain: Caddy terminates TLS, Anubis verifies the client (DENY known AI scrapers, CHALLENGE everyone -else), and accepted traffic returns to Caddy's internal listener on `:8016`, +else), and accepted traffic returns to Caddy's internal listener on `:8016` which proxies every request whole to the Authentik server with the Host header preserved; Authentik's embedded outpost routes on that Host against each proxy provider's `external_host`, serves the login flow in place for @@ -67,12 +67,12 @@ graph TB ``` Guarded (Anubis `:8923` → Caddy `:8016` → Authentik outpost → Caddy `:8017`): -homepage (`www.`), search, **grafana** (`dankclaw/Caddyfile:128-129`), +homepage (`www.`), search, **grafana** (`dankclaw/Caddyfile:128-129`) prometheus (`:135-137`), loki (`:141-143`), seerr (`:148-150`), marreta (`:155-160`), romm (`:167-169`), files (`:172-174`). Exempt / own-auth: `auth.` (the IdP login itself), `ai.` (OpenWebUI, its own OIDC), gitea (git smart-HTTP clients cannot solve JS challenges — `/info/refs` and `git-*-pack` pass -unchallenged), pihole (Caddy `basicauth`, `:167`), plex (plain HTTP), +unchallenged), pihole (Caddy `basicauth`, `:167`), plex (plain HTTP) books/jellyfrank (own logins), `hermes.`/`hermesadmin.`, and the *arr vhosts that terminate on FrankBook-Pro (`:254-281`, "exempt by construction: own auth"). Grafana is **not** exempt even though it signs in through authentik as @@ -104,13 +104,13 @@ Caddy holds one Let's Encrypt certificate per vhost, issued and renewed through `Network=searxng.network` (runtime network `systemd-searxng`), while `searxng.container`, `marreta.container` and `marreta-browser.container` join the tunnel the modern way — `Network=gluetun.container` -(`dankclaw/searxng/quadlet/searxng.container:18`, -`dankclaw/marreta/quadlet/marreta.container:34`, +(`dankclaw/searxng/quadlet/searxng.container:18` +`dankclaw/marreta/quadlet/marreta.container:34` `dankclaw/marreta/quadlet/marreta-browser.container:25`), the Quadlet equivalent of the retired compose `network_mode: service:gluetun`. **Five** containers share the VPN exit, not three, and Marreta's outbound fetches leave through the same WireGuard tunnel as SearXNG's. Pi-hole is a different kind of -exception (`Network=slirp4netns`, `dankclaw/pihole/quadlet/pihole.container`); +exception (`Network=slirp4netns`, `dankclaw/pihole/quadlet/pihole.container`) everything else on Dankclaw uses the normal LAN route. This matches KB Pattern 2 in [[Gluetun]]. See [[Container Network Namespaces]] for which addresses are reachable from that diff --git a/serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md b/serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md index eaa5769..c447265 100644 --- a/serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md +++ b/serverAppConfigs-docs/Architecture/Quadlet Lifecycle.md @@ -36,14 +36,14 @@ done ## 2. Cutover: compose service → Quadlet unit Checklist (precedents to copy: -`dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/`, +`dankclaw/{caddy,gitea,searxng,open-webui,seerr}/quadlet/` `dankclaw/observability/quadlet/`): 1. Unit file at `//quadlet/.container`, mirroring compose line for line: `Image=` (`localhost/` prefix for locally built images + `Pull=never`), `ContainerName=` (Caddy and scrape configs address containers by name), `Network=`, `User=0:0` when the image's own user - cannot write a rootless-owned bind mount, `Volume=` (`:ro` preserved), + cannot write a rootless-owned bind mount, `Volume=` (`:ro` preserved) `Exec=` for compose `command:`, `PublishPort=`, health keys, `[Service] Restart=always` + `RestartSec=10`, `[Install] WantedBy=default.target`. **Never `systemctl --user enable` the generated @@ -56,7 +56,7 @@ Checklist (precedents to copy: `systemctl --user restart .service` → `wait_for_health`; retire any legacy hand-written unit into `~/.AGENTTRASH/` (a leftover `~/.config/systemd/user/.service` that shells out to - `podman compose up` collides BY NAME with the generated unit); + `podman compose up` collides BY NAME with the generated unit) `podman rm -f ` so the unit's container can take the name. 4. Planner: map `//**` → service in `scripts/deploy-plan.py` AND add a `service_is_allowed` entry; keep `SERVICE_ORDER` dependencies @@ -123,7 +123,7 @@ the restart policy faithfully resurrects it forever. Quickest unblock: `podman compose up -d --force-recreate `. Durable fix = the §2 cutover. **Verify the capability, not the status:** `podman exec - python3 -c "import torch; print(torch.cuda.is_available())"`; + python3 -c "import torch; print(torch.cuda.is_available())"` `ss -ltn | grep ` for the actual bind addresses. ComfyUI incident: 2026-09-16, stale device-less container, 9,146 restarts. History repeats: 2026-08-26 Caddy (no unit), 2026-09-11 five observability services. @@ -143,7 +143,7 @@ Quickest unblock: `podman compose up -d --force-recreate `. Durable fix unhealthy but logs look fine, compare the baked `{{.Config.Healthcheck.Test}}` array against the intended command before believing the status. -- **Probe a new key before shipping:** drop a throwaway `probe.container`, +- **Probe a new key before shipping:** drop a throwaway `probe.container` `daemon-reload`, `journalctl --user --since "1 min ago" -o cat | grep -i "unsupported key"`, `systemctl --user cat probe.service` (shows the generated ExecStart verbatim), then remove + reload. diff --git a/serverAppConfigs-docs/Architecture/Testing Policy.md b/serverAppConfigs-docs/Architecture/Testing Policy.md index bf02032..7f132ce 100644 --- a/serverAppConfigs-docs/Architecture/Testing Policy.md +++ b/serverAppConfigs-docs/Architecture/Testing Policy.md @@ -54,4 +54,3 @@ Tests are written first. A failing test means the seam is not satisfied; the con ## See Also - [[Deployment Flow]] -- [[index]] diff --git a/serverAppConfigs-docs/Dankclaw/ComfyUI Qwen-Image Workflows.md b/serverAppConfigs-docs/Dankclaw/ComfyUI Qwen-Image Workflows.md index 97a8f8b..e36aebe 100644 --- a/serverAppConfigs-docs/Dankclaw/ComfyUI Qwen-Image Workflows.md +++ b/serverAppConfigs-docs/Dankclaw/ComfyUI Qwen-Image Workflows.md @@ -56,8 +56,8 @@ sampling), `sampler_name: euler`, `scheduler: simple`, `denoise: 1.0`, seed fixed in the file (`1930` in the art workflow, `2026` in the text one) — read from `dankclaw/comfyui/workflows/*.json` node `9`. -Node `class_type`s in both files: `CLIPTextEncode` ×2, `UNETLoader`, -`CLIPLoader`, `VAELoader`, `EmptySD3LatentImage`, `KSampler`, `VAEDecode`, +Node `class_type`s in both files: `CLIPTextEncode` ×2, `UNETLoader` +`CLIPLoader`, `VAELoader`, `EmptySD3LatentImage`, `KSampler`, `VAEDecode` `SaveImage` (ids 3–11). | Workflow | File | Shape | Steps | Made for | @@ -113,9 +113,9 @@ only `models`, `input`, `output`, `custom_nodes` and the healthcheck file Quantized preferred, sized for the 8060S iGPU (GTT-backed — `/system_stats` reports 122.4 GB of device VRAM on `cuda:0`, `curl http://127.0.0.1:8188/system_stats` 2026-10-18; the RTX 3090 is frank-zealot's). All repos verified 2026-10-05 and -all four still resolve today (`huggingface.co/lightx2v/Qwen-Image-Lightning`, -`huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF`, -`huggingface.co/Comfy-Org/Qwen-Image-DiffSynth-ControlNets`, +all four still resolve today (`huggingface.co/lightx2v/Qwen-Image-Lightning` +`huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF` +`huggingface.co/Comfy-Org/Qwen-Image-DiffSynth-ControlNets` `huggingface.co/QuantStack/Wan2.2-TI2V-5B-GGUF`, 2026-10-18). None are on disk yet: `models/` holds no lora/, controlnet/ or Wan/Edit file (`ls -R` 2026-10-18). @@ -126,7 +126,7 @@ yet: `models/` holds no lora/, controlnet/ or Wan/Edit file (`ls -R` 2026-10-18) | 3 | **DiffSynth ControlNets** (structure control) | `Comfy-Org/Qwen-Image-DiffSynth-ControlNets` | `split_files/model_patches/` canny + depth | ~2–3 GB each | Pose/edge/depth-guided layouts — pairs with the legible-text workflow for exact poster composition. The `QwenImageDiffsynthControlnet` node is already in the running ComfyUI (`GET http://127.0.0.1:8188/object_info/QwenImageDiffsynthControlnet` → 200, 2026-10-18) | | 4 | **Wan 2.2 TI2V-5B GGUF** (first video model) | `QuantStack/Wan2.2-TI2V-5B-GGUF` | Q4/Q5 GGUF (~6–8 GB) | ~6–8 GB | T2V+I2V in one dense 5B model; the gentlest on-ramp into video generation, runs in ComfyUI core (`wan` is in this build's `CLIPLoader` type enum — `GET http://127.0.0.1:8188/object_info/CLIPLoader` 2026-10-18) | -Placement: #1 → `models/lora/` (create it), #2 → `models/diffusion_models/`, +Placement: #1 → `models/lora/` (create it), #2 → `models/diffusion_models/` #3 → `models/controlnet/` (create it), #4 → `models/diffusion_models/`. Fetch with `scripts/download-frank-*.sh` discipline (`-f` + byte-count + sha256 checks) or a plain `hf download` with the token in the environment — diff --git a/serverAppConfigs-docs/Dankclaw/ComfyUI.md b/serverAppConfigs-docs/Dankclaw/ComfyUI.md index 5681527..b2dfd25 100644 --- a/serverAppConfigs-docs/Dankclaw/ComfyUI.md +++ b/serverAppConfigs-docs/Dankclaw/ComfyUI.md @@ -43,7 +43,7 @@ Dankclaw is an AMD Strix Halo APU (gfx1151). Standard PyTorch ROCm wheels do not ## GPU device access (rootless), and why the unit exists -The unit passes `/dev/kfd` and `/dev/dri` (`AddDevice=`, +The unit passes `/dev/kfd` and `/dev/dri` (`AddDevice=` `dankclaw/comfyui/quadlet/comfyui.container:45-46`) and uses `GroupAdd=keep-groups`. A literal `render` entry cannot work: rootless Podman cannot map the group name into the user namespace. The companion udev rule @@ -115,7 +115,7 @@ generated units. The web UI responds on `/` (200 `text/html`, `curl http://127.0.0.1:8188/` 2026-10-18). ComfyUI also exposes `/system_stats`. On this APU, `/system_stats` reports `device=cuda:0 Radeon 8060S Graphics` (type `cuda`, the ROCm device). The -health probe is the repo file `dankclaw/comfyui/comfyui-healthcheck.py`, +health probe is the repo file `dankclaw/comfyui/comfyui-healthcheck.py` bind-mounted to `/usr/local/bin/comfyui-healthcheck.py` and run as `HealthCmd` (`dankclaw/comfyui/quadlet/comfyui.container:52,57`); it GETs `http://127.0.0.1:8188/` inside the container, and the container reports @@ -135,5 +135,5 @@ bind-mounted to `/usr/local/bin/comfyui-healthcheck.py` and run as `HealthCmd` - [[Dankclaw Stack]] - [[ComfyUI ROCm and Vulkan]] — the live-verified GPU path research - [[ComfyUI Qwen-Image Workflows]] — the Qwen-Image 2.1 payload, proven workflows, and recommended model/LoRA additions -- [[Lemonade and sd.cpp]] — the ggml-family alternative +- the ggml-family alternative - Knowledge base: [[ComfyUI]] (tools) diff --git a/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md b/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md index 9123e12..093670f 100644 --- a/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md +++ b/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md @@ -52,11 +52,10 @@ graph LR The LLM rows are bare-metal systemd services, not containers, and this repo owns their scripts, units and knowledge base (see -[LLM fleet knowledge base](../LLM/index.md)). Which payload and runtime sits on +). Which payload and runtime sits on each port is era state and lives on the endpoint page, not here. `frank-npu` (the XDNA2 NPU endpoint) and `frank-medium` (the predecessor of `frank-archon` -on the Dankserver port) are both retired — see [[frank-npu]] and -[[frank-medium-radiance-mxfp4]]. +on the Dankserver port) are both retired — see and ## Boot Persistence @@ -72,7 +71,7 @@ Deploy the SearXNG trio with `scripts/deploy-searxng-quadlet.sh dankclaw`, open- ## See Also -- [[index]] — repository index +- repository index - [[Network Architecture]] — routing and proxy - [[Observability]] — Grafana, Prometheus, logs -- [LLM fleet knowledge base](../LLM/index.md) — the model-serving endpoints on this host +- the model-serving endpoints on this host diff --git a/serverAppConfigs-docs/Dankclaw/Frank TTS Engine.md b/serverAppConfigs-docs/Dankclaw/Frank TTS Engine.md index 981ea19..8ff5ad8 100644 --- a/serverAppConfigs-docs/Dankclaw/Frank TTS Engine.md +++ b/serverAppConfigs-docs/Dankclaw/Frank TTS Engine.md @@ -37,7 +37,7 @@ timestamp: 2026-10-18 ## Why this engine, and not the alternatives -The criterion Frank set on 2026-10-18 was **"whatever is better for live output"**, and live output is decided by time-to-first-audio and by whether the stream stays ahead of real time — not by benchmark throughput. RTF only has to stay under 1.0 with margin; what separates the arms is first-byte latency and **spread**. Measured on this host the same day (CPU; the full A/B is the section *"Decision: arm A for live output (2026-10-18)"* in [[Local TTS and Voice Cloning]], which is the source of these numbers): +The criterion Frank set on 2026-10-18 was **"whatever is better for live output"**, and live output is decided by time-to-first-audio and by whether the stream stays ahead of real time — not by benchmark throughput. RTF only has to stay under 1.0 with margin; what separates the arms is first-byte latency and **spread**. Measured on this host the same day (CPU; the full A/B is the section *"Decision: arm A for live output (2026-10-18)"* in , which is the source of these numbers): | Measure | Resident `tts-server` (arm A, chosen) | Mainline per-request CLI path (arm B) | |---|---|---| @@ -128,6 +128,6 @@ still open. | Everything is ~2× slower after a systemd tweak | a cpuset/cap pin changed `hardware_concurrency()` and the engine resized its thread pools | remove the pin (law 2) | | `502` from the façade with `frank-tts upstream request failed` | the façade cannot resolve or reach the engine | `podman network inspect systemd-frank-tts-engine` and `podman inspect frank-tts --format '{{json .NetworkSettings.Networks}}'` — the façade must be attached to the bridge; a `127.0.0.1:8880` or `host.containers.internal:8880` upstream in `scripts/frank-tts.env` is the old shape and cannot work | | A voice is missing from the list | its directory has no `reference.wav`, or the name is not lowercase | fix the pack; both are refused loudly by design | -| `podman exec` in the deploy says `no such file or directory` for `/app/warm-voices.sh` | the image predates the warmer | re-deploy: since 2026-10-18 `build_compose_image` hashes every file the image `COPY`s, not only `Containerfile.engine`, so editing the warmer rebuilds it — see [[deployment]] | +| `podman exec` in the deploy says `no such file or directory` for `/app/warm-voices.sh` | the image predates the warmer | re-deploy: since 2026-10-18 `build_compose_image` hashes every file the image `COPY`s, not only `Containerfile.engine`, so editing the warmer rebuilds it — see | | The warm pass logs `registered, but the warm synthesis failed` for EVERY voice, while a hand-run `/app/warm-voices.sh` succeeds | `WARM_TEXT` arrived **with its quotes**. podman's `--env-file` does no shell parsing — it takes the literal remainder of the line — so the env-file reader got `"Warm up, one two three."` where `source` gives `Warm up, one two three.`; the quotes plus a backslash then make the synthesis body invalid JSON | the hook strips one matched pair and escapes what is left; prove the value with `podman exec frank-tts-engine env \| grep WARM_TEXT`. The seam pins both readers, because a test that only sources the env file cannot see this | | A `502` for `/v1/voices` where the engine answers `200` on `/health` from the same boot | the engine restarted and its in-memory registry is empty again — `/health` is model-loaded, NOT voices-registered | re-run `podman exec frank-tts-engine /app/warm-voices.sh`; that is the same distinction the `frank-tts` healthcheck enforces | diff --git a/serverAppConfigs-docs/Dankclaw/Frank TTS.md b/serverAppConfigs-docs/Dankclaw/Frank TTS.md index b0344c0..4f722fa 100644 --- a/serverAppConfigs-docs/Dankclaw/Frank TTS.md +++ b/serverAppConfigs-docs/Dankclaw/Frank TTS.md @@ -18,10 +18,10 @@ audio in that cloned voice. Everything else in the voice stack sits behind it the engine ([[Frank TTS Engine]]) publishes no port on purpose, and consumers never learn a path. -**Deployed 2026-10-18** — `./scripts/deploy.sh dankclaw --service frank-tts`, +**Deployed 2026-10-18** — `./scripts/deploy.sh dankclaw --service frank-tts` both units `active`, no drop-ins, and the homepage card's `wip` gate came off in the same change. Probes on the deployed endpoint: `/health` answers -`{"status":"ok","backend":"omni-http","voices":["narrator1","narrator2","narrator3"]}`, +`{"status":"ok","backend":"omni-http","voices":["narrator1","narrator2","narrator3"]}` and two different voices rendered the same sentence into **different** bytes (24 kHz mono PCM16, 4.24 s, healthy RMS) — so name resolution is real, not a pass-through of one default. The operator harness routes its `speech` role here; @@ -147,7 +147,7 @@ A `/health` probe job would therefore be structurally never-green, and because `LlmEndpointDown` is scoped across the whole `frank-*` job family (the rule set lives on [[Prometheus]]), a permanently red `frank-tts` target would page on Signal every few minutes for an endpoint that is fine. The engine cannot be -scraped at all: it publishes nothing. Status lives in `systemctl --user status`, +scraped at all: it publishes nothing. Status lives in `systemctl --user status` `podman healthcheck run`, the two unit journals, and the homepage card, which probes `127.0.0.1:7772` for real now — the `wip: down` gate came off with the 2026-10-18 deploy. @@ -155,7 +155,7 @@ probes `127.0.0.1:7772` for real now — the `wip: down` gate came off with the ## The harness speech role The operator harness reaches this endpoint through -`~/.omp/agent/models.yml` — provider `frank-tts (dankclaw)`, +`~/.omp/agent/models.yml` — provider `frank-tts (dankclaw)` `baseUrl: http://192.168.0.143:7772/v1`, `auth: none`, and the runner api `openai-speech` on the **model** row (naming a runner api at *provider* level fails the whole file's validation and disables every custom provider, which is @@ -171,7 +171,7 @@ harness bundle and measured the day of the flip: `speech.voice` is the TUI's **speak-aloud** voice — its enum is Kokoro's list, and it never reaches an HTTP request. So a harness call arrives voice-less by default, which is exactly why the endpoint grew `DEFAULT_VOICE`: the declared - pack answers a voice-less request and the journal marks that line `(default)`, + pack answers a voice-less request and the journal marks that line `(default)` while an **unknown** name still 400s with the list that exists. - **`speech.voice` must stay a Kokoro name.** Setting it to a pack name does not select a pack; it hands the vocalizer a value outside its enum. (The first diff --git a/serverAppConfigs-docs/Dankclaw/Gitea.md b/serverAppConfigs-docs/Dankclaw/Gitea.md index dac5aad..fa20d5a 100644 --- a/serverAppConfigs-docs/Dankclaw/Gitea.md +++ b/serverAppConfigs-docs/Dankclaw/Gitea.md @@ -33,7 +33,7 @@ The containers share the private bridge network `systemd-gitea`. The quadlet net | `dank/command-center` | second public repo on the instance (`GET http://127.0.0.1:3000/api/v1/repos/search` 2026-10-18) | | `dank/vllmScripts` | retired 2026-08-22 — merged into `dank/serverAppConfigs` (history preserved) *(unverified 2026-10-18: the anonymous API search returns no match and `/dank/vllmScripts` is 404, so whether the repo still exists privately cannot be checked without a token)* | -The LLM fleet knowledge base: [serverAppConfigs-docs/LLM/index.md](../LLM/index.md) +The LLM fleet knowledge base: ## Management diff --git a/serverAppConfigs-docs/Dankclaw/Hermes-Dashboard.md b/serverAppConfigs-docs/Dankclaw/Hermes-Dashboard.md index 2ca8d6b..0cd8b33 100644 --- a/serverAppConfigs-docs/Dankclaw/Hermes-Dashboard.md +++ b/serverAppConfigs-docs/Dankclaw/Hermes-Dashboard.md @@ -30,7 +30,7 @@ entrypoint does not start the dashboard. The dashboard is a separate process, so it runs in its own Quadlet unit ([`dankclaw/hermes/quadlet/hermes-dashboard.container`](../../dankclaw/hermes/quadlet/hermes-dashboard.container): image `localhost/hermes-agent:latest`, `Pull=never`, `Exec=hermes dashboard ---host 0.0.0.0 --port 9119 --no-open --skip-build`, `Restart=always`, +--host 0.0.0.0 --port 9119 --no-open --skip-build`, `Restart=always` `RestartSec=5`). All three Hermes containers mount the same `/data` bind mount. Unlike its two siblings — `hermes.container` and `hermes-webui.container` both diff --git a/serverAppConfigs-docs/Dankclaw/Hermes-WebUI.md b/serverAppConfigs-docs/Dankclaw/Hermes-WebUI.md index f790365..3e57e20 100644 --- a/serverAppConfigs-docs/Dankclaw/Hermes-WebUI.md +++ b/serverAppConfigs-docs/Dankclaw/Hermes-WebUI.md @@ -40,7 +40,7 @@ from another LAN host: `/health` → 200, `/` → 302) — **Vhost:** ## Why a container, not a host process -The gateway's state files are owned by the container-mapped uid (`109999`, +The gateway's state files are owned by the container-mapped uid (`109999` from rootless Podman's `1 → 100000/65536` subuid range — `/etc/subuid` → `dank:100000:65536`, and the agent's runtime uid is 10000, so 100000 + 9999 = 109999; `ls -ld /home/dank/Containers/hermes/data` → uid 109999, 2026-10-18). A fresh rootless @@ -82,12 +82,12 @@ the sibling dashboard on `9119` is not. - Dedicated `tunnel` account on Dankclaw (`getent passwd tunnel` → uid 994, shell `/usr/sbin/nologin`, 2026-10-18; password locked). Authentication is key-only - *(unverified 2026-10-18: `/home/tunnel/.ssh/authorized_keys` is not readable + - (unverified 2026-10-18: `/home/tunnel/.ssh/authorized_keys` is not readable without `sudo`)*; the account can do nothing except authenticate — `ForceCommand /usr/sbin/nologin` kills any session channel while the transport stays up for port forwarding. - sshd drop-in `/etc/ssh/sshd_config.d/99-tunnel.conf` (exists, 118 B, 2026-08-31) - scopes `AllowTcpForwarding yes` to that user only, plus `PermitTTY no`, + scopes `AllowTcpForwarding yes` to that user only, plus `PermitTTY no` `X11Forwarding no` and `ForceCommand /usr/sbin/nologin` — all four lines are present under `Match User tunnel` (`cat` 2026-10-18). - Key options on the authorized key pin the blast radius: @@ -130,7 +130,7 @@ only if inactive (it is never restarted on deploy, so in-flight browser sessions survive), re-tags the image alias, then restarts `hermes.service`, `hermes-dashboard.service` and `hermes-webui.service` in the **user** manager of `dank` (linger must be on for -them to come up at boot — `loginctl show-user dank -p Linger` → `yes`, +them to come up at boot — `loginctl show-user dank -p Linger` → `yes` 2026-10-18), so gateway/dashboard/webui restarts stay isolated from inference deploys. Before first deploy, set a strong `HERMES_WEBUI_PASSWORD` in the live diff --git a/serverAppConfigs-docs/Dankclaw/Hermes.md b/serverAppConfigs-docs/Dankclaw/Hermes.md index 461ab30..bebf1d9 100644 --- a/serverAppConfigs-docs/Dankclaw/Hermes.md +++ b/serverAppConfigs-docs/Dankclaw/Hermes.md @@ -18,7 +18,7 @@ bindings, 2026-10-18). Note the **container name drift**: the unit is `podman run --name systemd-%N …` (`systemctl --user cat hermes.service`), so `podman ps` shows the gateway as **`systemd-hermes`** — the repo unit sets no `ContainerName=` key even though its header comment says "container name: -hermes". The user-facing web surface is [[Hermes-WebUI]] on `:8787`, reached via +hermes". The user-facing web surface is [[Hermes-WebUI]] on `:8787`, reached `hermes.franksdank.cloud` — internal-only by DNS absence (no public Porkbun A record; the name resolves from the Pi-hole hosts file, `dankclaw/pihole/addn-hosts:39`) — **Docs:** @@ -37,7 +37,7 @@ designed wiring; the remedy for both halves is in the log line **Persistent state:** `/home/dank/Containers/hermes/data` on Dankclaw, bind-mounted at `/data` in all three Hermes containers (gateway, webui, dashboard — -`hermes.container:36`, `hermes-webui.container:46`, +`hermes.container:36`, `hermes-webui.container:46` `hermes-dashboard.container:24`). Each container additionally gets an anonymous volume at `/opt/data`, which is re-created empty on every container recreation — anything that lives only there @@ -80,7 +80,7 @@ is not durable. The tracked, sanitized configuration baseline is `frank-dragoon` — verified from the host with `curl http://127.0.0.1:7770/v1/models` → a single served id (2026-10-18; write `127.0.0.1`, not `localhost`, because that port is a rootless pasta publish - and is IPv4-only); + and is IPv4-only) `dankserver-small` reaches `http://192.168.0.235:7770/v1` and, **as of 2026-10-18, that endpoint serves only `frank-archon`** (`GET http://192.168.0.235:7770/v1/models` → `[{"id":"frank-archon"}]`). The name @@ -115,7 +115,7 @@ per session; the browser itself lives in a dedicated daemon container, the digest-pinned headless-shell unit `chrome-devtools-protocol.container` (host network, loopback-only bind on 127.0.0.1:9222 — `ss -ltnp` 2026-10-18 shows `127.0.0.1:9222 headless-shell` and nothing on `0.0.0.0`; image -`docker.io/chromedp/headless-shell@sha256:2d349b544a1ea6b5…`, +`docker.io/chromedp/headless-shell@sha256:2d349b544a1ea6b5…` `dankclaw/hermes/quadlet/chrome-devtools-protocol.container:28`). Both consumers — the Signal gateway and the webui server — run in the host network namespace (`podman inspect` → `net=host` for both, 2026-10-18), so they can @@ -248,7 +248,7 @@ What actually breaks, and the fix: effort: none # mandatory, see below ``` - Before copying this: which id is served on which port is endpoint state — see + Before copying this: which id is served on which port is endpoint state — [[frank-zealot]] and [[frank-archon]]. The provider/model pairing above is the shape that was landed in 2026; re-derive it from the live config before reusing it. @@ -282,7 +282,7 @@ Auth contract (verified live 2026-08-31, re-checked 2026-10-18): `POST /api/auth on match, 401 on miss. No username field. `/health` answers 200 unauthenticated (`curl http://127.0.0.1:8787/health` → 200 with `{"status":"ok", …}` today) and `/` answers 302 (`curl -o /dev/null -w -%{http_code} http://127.0.0.1:8787/` → 302 today) — **never verify auth via +%{http_code} http://127.0.0.1:8787/` → 302 today) — **never verify auth those**; probe login with a deliberately wrong password (expect 401) and the real one (expect 200). Details on the webui surface: [[Hermes-WebUI]]. @@ -311,7 +311,7 @@ login with a deliberately wrong password (expect 401) and the real one instead of loopback" — live bind `0.0.0.0:8787`, `ss -ltnp` 2026-10-18; off-network ingress is SSH tunnel only — user `tunnel`, port **22** (`sshd -T` → `port 22`, 2026-10-18), key-only auth - *(unverified 2026-10-18: the per-user `Match` block needs `sudo` to read)*, + - (unverified 2026-10-18: the per-user `Match` block needs `sudo` to read)* forward remote end localhost:8787). diff --git a/serverAppConfigs-docs/Dankclaw/Marreta.md b/serverAppConfigs-docs/Dankclaw/Marreta.md index 308a056..0ff243d 100644 --- a/serverAppConfigs-docs/Dankclaw/Marreta.md +++ b/serverAppConfigs-docs/Dankclaw/Marreta.md @@ -19,7 +19,7 @@ returns a stripped, readable version of the page — no paywall chrome, no tracking, no cookie walls. It is a Laravel 13 app on the `serversideup/php:8.4-fpm-nginx` image (`composer.json` requires `php: ^8.4`, `laravel/framework: ^13.0`, `filament/filament: ^5.7` — -), +) with an optional headless-browser companion for JavaScript-heavy pages. **Host:** Dankclaw — **vhost:** `marreta.franksdank.cloud` (Caddy, guard-chained) @@ -31,7 +31,7 @@ with an optional headless-browser companion for JavaScript-heavy pages. - Fetches a target URL with a cascade of strategies: plain HTTP (curl), a headless browser (Lightpanda) for JS pages, and the Internet Archive (Wayback) as a last resort — the three `app/Services/Fetch/*FetchStrategy.php` - files upstream (`BrowserFetchStrategy`, `CurlFetchStrategy`, + files upstream (`BrowserFetchStrategy`, `CurlFetchStrategy` `WaybackFetchStrategy`, dispatched by `app/Services/FetchManager.php`). - Strips paywall/consent/tracking markup and re-renders clean HTML. - Ships a Filament admin panel at `/admin` (login = `ADMIN_EMAIL` / @@ -69,7 +69,7 @@ graph LR | `marreta-browser.service` | `dankclaw/marreta/quadlet/marreta-browser.container` | `docker.io/lightpanda/browser:nightly` (`:26`) | `gluetun.container` (`:25`) | `9222` (CDP) | Both units are boot-persistent (`WantedBy=default.target`) and self-healing -(`Restart=always`, `RestartSec=10` — `marreta.container:62-69`, +(`Restart=always`, `RestartSec=10` — `marreta.container:62-69` `marreta-browser.container:30-34`), per the repo's Boot Persistence rule. Live 2026-10-18 both report `systemctl --user is-enabled` → `generated` and `is-active` → `active`, and `podman inspect` reports marreta **healthy** @@ -87,14 +87,14 @@ The fix is a single env var, verified against the serversideup source: - The image sets `ENV NGINX_HTTP_PORT="8080"` and the site template is `listen ${NGINX_HTTP_PORT} default_server;`. -- The image does **not** bake `conf.d/*.conf` (it `rm -rf`s them at build); +- The image does **not** bake `conf.d/*.conf` (it `rm -rf`s them at build) the entrypoint `10-init-webserver-config.sh` runs `envsubst` over every `NGINX_*` variable **at container start**. - So `Environment=NGINX_HTTP_PORT=8085` in the quadlet relocates the listen port off the collision with no custom build. Marreta then publishes **nothing** of its own — the netns owner (Gluetun) -publishes `8082:8085` on its behalf (`dankclaw/searxng/quadlet/gluetun.container:32`), +publishes `8082:8085` on its behalf (`dankclaw/searxng/quadlet/gluetun.container:32`) exactly as it publishes `8081:8080` for SearXNG. Caddy dials `192.168.0.143:8082` (`dankclaw/Caddyfile:419-422`). Live env confirms the relocation: `podman inspect marreta` carries `NGINX_HTTP_PORT=8085` @@ -127,7 +127,7 @@ on the mechanism: `BrowserFetchStrategy.php` only calls (`dankclaw/Caddyfile:159-163`). 2. **Auth decision** — Anubis dials Caddy's `http://:8016`, which proxies whole-request to `authentik-server:9000` (`dankclaw/Caddyfile:22-24,94`). -3. **Post-auth routing** — authenticated traffic returns to `http://:8017`, +3. **Post-auth routing** — authenticated traffic returns to `http://:8017` where `@marreta host marreta.franksdank.cloud` routes to `reverse_proxy 192.168.0.143:8082` (`dankclaw/Caddyfile:419-422`). @@ -164,7 +164,7 @@ Marreta ships `trustProxies(at: '172.16.0.0/12,10.0.0.0/8,192.168.0.0/16,127.0.0 — that is the `TRUSTED_PROXIES` fallback in upstream `bootstrap/app.php`, and this container sets no `TRUSTED_PROXIES`, so the default is what runs (`podman inspect marreta` 2026-10-18) — and forces `https` when `APP_URL` -starts with `https://` (`app/Providers/AppServiceProvider.php:23-24`, +starts with `https://` (`app/Providers/AppServiceProvider.php:23-24` `URL::forceScheme('https')`). With `APP_URL=https://marreta.franksdank.cloud` (`dankclaw/marreta/quadlet/marreta.container:41`) the app honors the forwarded scheme and client IP through the whole chain, so there is no redirect loop @@ -234,7 +234,7 @@ netns down and would otherwise orphan it). Deployed via `deploy.sh dankclaw --service gluetun --service marreta --service caddy` (gluetun first to recreate the tunnel with the new `8082:8085` publish, which also cycles SearXNG's netns ref). Authentik `ProxyProvider` `marreta` -registered (pk 13, `external_host=https://marreta.franksdank.cloud/`, +registered (pk 13, `external_host=https://marreta.franksdank.cloud/` `internal_host=http://caddy:8017`). Results: - Unauth edge → Anubis challenge page (not 502). TLS cert issued via DNS-01. @@ -245,14 +245,14 @@ registered (pk 13, `external_host=https://marreta.franksdank.cloud/`, (`GET 127.0.0.1:9222/json/version` → `Lightpanda/1.0`). - SearXNG unaffected: `:8081` → 200 after the tunnel recreate. -> **Gotcha hit during deploy:** the first seed used a bare-base64 `APP_KEY`, +> **Gotcha hit during deploy:** the first seed used a bare-base64 `APP_KEY` > which 500s every app route (see Secrets above). Fixed by re-keying with the > `base64:` prefix before any encrypted data was persisted. ## Citations - Marreta upstream: (README, - `compose.yml`, `bootstrap/app.php`, `app/Services/Fetch/BrowserFetchStrategy.php`, + `compose.yml`, `bootstrap/app.php`, `app/Services/Fetch/BrowserFetchStrategy.php` `lang/` directory listing). - serversideup/php fpm-nginx port mechanism: `serversideup/docker-php` `src/variations/fpm-nginx/Dockerfile` (`NGINX_HTTP_PORT="8080"`) and diff --git a/serverAppConfigs-docs/Dankclaw/OpenWebUI.md b/serverAppConfigs-docs/Dankclaw/OpenWebUI.md index 42e277d..5e6d7e6 100644 --- a/serverAppConfigs-docs/Dankclaw/OpenWebUI.md +++ b/serverAppConfigs-docs/Dankclaw/OpenWebUI.md @@ -111,7 +111,7 @@ The Ollama connection is disabled (`ollama.enable=false`). - Working dir for imports: `/app/backend`; package name `open_webui` (underscore), NOT `openwebui`. `os.environ["WEBUI_SECRET_KEY"]` MUST be set - BEFORE importing any `open_webui` module (import-time hard requirement); + BEFORE importing any `open_webui` module (import-time hard requirement) the secret lives at `/app/backend/.webui_secret_key`. - **The `sqlite3` CLI is NOT in the container** — use Python's `sqlite3` module (`podman exec open-webui python3 -c "import sqlite3; @@ -122,8 +122,8 @@ The Ollama connection is disabled (`ollama.enable=false`). pydantic `ModelModel` instances (attribute access, not `.get()`), with NO `status`/`enabled` attribute — use `is_active` from the DB. - **API auth for verification:** `create_token` lives in - `open_webui.utils.auth` (NOT `open_webui.utils.utils`); - `get_verified_user` requires an `id` claim (not just `sub`); + `open_webui.utils.auth` (NOT `open_webui.utils.utils`) + `get_verified_user` requires an `id` claim (not just `sub`) `SESSION_SECRET = WEBUI_SECRET_KEY`. Mint an admin token inside the container: @@ -148,7 +148,7 @@ The Ollama connection is disabled (`ollama.enable=false`). `ClientConnectorError`: a network-reachability problem (host LAN IP vs `host.containers.internal`), not a model problem. -**Durability:** the model table + connections are DB-only (not git-managed); +**Durability:** the model table + connections are DB-only (not git-managed) a fresh data volume loses them — including the public access grant above. For reproducibility, add an idempotent first-start seed script or document the setup in the service doc. @@ -156,7 +156,7 @@ the setup in the service doc. ## API access (auth-gated) Every `/api/*` endpoint requires an authenticated session (SSO login JWT) -or a per-user API key (`auth.enable_api_keys=true`; keys are minted per +or a per-user API key (`auth.enable_api_keys=true`; keys are minted account under Profile → API Keys). Anonymous web visitors get the `pending` role (`ui.default_user_role=pending`), which cannot chat. Verified 2026-10-07 from loopback `:3005` and over the public vhost: @@ -166,13 +166,12 @@ bogus bearer token); only `/api/config`, `/api/version` and `/health` `POST /api/chat/completions` all return **401** with no key (also with a bogus bearer token); only `/api/config`, `/api/version` and `/health` answer publicly, and none of them expose the model list. An external program needs an account + API key — there is -no keyless access. Re-verified 2026-10-18 from loopback: `/health` → `{"status":true}`, +no keyless access. Re-verified 2026-10-18 from loopback: `/health` → `{"status":true}` `/api/version` → 0.11.4, `/api/v1/models/list`, `/api/v1/models` and `POST /api/chat/completions` → 401; `/api/config` additionally publishes `features.enable_login_form=false`, `features.enable_signup=false` and `oauth.providers.oidc="authentik"` — the SSO-only posture, visible from outside. - ## Web search Chat web search is served by [[SearXNG]] on this host, wired in the quadlet @@ -186,7 +185,7 @@ Chat web search is served by [[SearXNG]] on this host, wired in the quadlet The Anubis→Authentik guard chain is a browser mechanism (proof-of-work + session flow); a plain HTTP client cannot solve it, and the internal port bypasses Caddy entirely. -- **Limiter:** SearXNG caps non-HTML (JSON) requests at 4/IP/hour (`API_MAX`, +- **Limiter:** SearXNG caps non-HTML (JSON) requests at 4/IP/hour (`API_MAX` a hardcoded Python constant — no config knob). The `pass_ip` list is checked FIRST and short-circuits the whole botdetection chain, and OpenWebUI's bridge egress (`10.89.x`) is in `pass_ip`, so the cap never applies. Verified: an @@ -249,7 +248,6 @@ browser-driven and only touches the two exempt vhosts (`ai.` and `auth.`). restore the password form: set `ui.enable_login_form` back to `true` in the Admin Panel (or DB) and drop the two env vars from the quadlet. - ## Boot persistence (Quadlet) The unit lives at `dankclaw/open-webui/quadlet/open-webui.container`. On the host it installs to `~/.config/containers/systemd/open-webui.container`. The generator turns it into `open-webui.service`. Because the unit sets `[Install] WantedBy=default.target` and user linger is on, it starts at boot. `systemctl --user is-enabled open-webui.service` reports `generated` (the quadlet boot-start state). @@ -296,7 +294,7 @@ TTS is the fleet's one OpenAI-compatible speech endpoint — `http://host.containers.internal:7772/v1` (`audio.tts.openai.api_base_url` in the config DB, 2026-10-18), model `frank-tts`, voice `narrator1`. That port is the rootless quadlet `frank-tts` (this repo: -`dankclaw/voiceservice/quadlet/frank-tts.container`, +`dankclaw/voiceservice/quadlet/frank-tts.container` `scripts/frank-tts.env`), and its shim resolves `voice=` against `/home/dank/voices//reference.wav` — a name without a reference is a 400 listing the voices that exist, never a silent fallback. @@ -310,7 +308,7 @@ setting). Its engine select takes `""` (Default) or `browser-kokoro`; when `browser-kokoro` is chosen the page synthesizes locally and rejects any voice outside its fixed pack — so `narrator1` fails with `Voice: "narrator1" not found. Should be one of: af_heart, …` **without a -single request reaching the server**. Personal audio settings are per +single request reaching the server**. Personal audio settings are browser: no `user.settings` row and no container env carries an audio override, yet the toast still appears ⇒ the choice lives in that browser's own storage. Fix in the UI: Settings → Audio → TTS Engine → **Default**, @@ -325,7 +323,6 @@ the failing click — that contrast is the instrument: a server-path failure leaves a log line and a frank-tts error, a `browser-kokoro` failure leaves neither. - ## Retired instances - The compose `open-webui` service (port 3000) was never run: 3000 is Gitea's HTTP port, so it could not bind. @@ -355,7 +352,7 @@ neither. began serving `frank-zealot` / `frank-dragoon` / `frank-archon`, so three id-keyed objects had to move in the same window as the alias flip: `model.id`+`name` (`frank-zealot` → `frank-zealot`, `params` - `{"context":{"length":200704}}` and the `meta.description` kept), + `{"context":{"length":200704}}` and the `meta.description` kept) `access_grant` (`resource_type='model'`, `resource_id`, principal `user`/`*`, permission `read`), and `openai.api_configs` per-backend `model_ids` (index 2 = `:7771` is the enabled family backend; indices 0/1 diff --git a/serverAppConfigs-docs/Dankclaw/Postgres.md b/serverAppConfigs-docs/Dankclaw/Postgres.md index e6b2a02..9a468e4 100644 --- a/serverAppConfigs-docs/Dankclaw/Postgres.md +++ b/serverAppConfigs-docs/Dankclaw/Postgres.md @@ -88,7 +88,7 @@ never enter this repo. See [[Gitea]] and [[Authentik]]. ## Boot persistence (Quadlet) -Both units carry `Restart=always` (`gitea-postgres.container:28`, +Both units carry `Restart=always` (`gitea-postgres.container:28` `authentik-postgres.container:38`) and `[Install] WantedBy=default.target` (`gitea-postgres.container:31`, `authentik-postgres.container:41-42`), are installed under `~/.config/containers/systemd/`, and report `generated` / `active` with diff --git a/serverAppConfigs-docs/Dankclaw/SearXNG Providers.md b/serverAppConfigs-docs/Dankclaw/SearXNG Providers.md index 8fe2e29..d439aa1 100644 --- a/serverAppConfigs-docs/Dankclaw/SearXNG Providers.md +++ b/serverAppConfigs-docs/Dankclaw/SearXNG Providers.md @@ -143,7 +143,7 @@ engines: disabled: false ``` -Key per-engine keys (each overwrites the engine-module default; see +Key per-engine keys (each overwrites the engine-module default `settings_engines.html`): | Key | Effect | @@ -164,7 +164,7 @@ Key per-engine keys (each overwrites the engine-module default; see *different* category. When the primary `google` gets CAPTCHA'd it suspends, but `google2` still returns results, and vice-versa. -**Parser caveat (read before fiddling with `parsers`):** the modern `google`, +**Parser caveat (read before fiddling with `parsers`):** the modern `google` `duckduckgo`, and `brave` engines ship a **custom `response()`** with hardcoded XPath and *ignore* the generic `parsers`/`p_parsers` table. So `parsers: [scrapy_html]` under `name: google` mostly does nothing. To make Google @@ -376,14 +376,14 @@ done **2. `/config` REST endpoint** lists every engine + enabled/disabled + categories — an inventory of what is live. It answers 200 on this instance today -(`curl http://127.0.0.1:8081/config`, 2026-10-18; each entry carries `enabled`, +(`curl http://127.0.0.1:8081/config`, 2026-10-18; each entry carries `enabled` `categories`, `timeout`, `shortcut`, and it is the source of the engine counts on [[SearXNG]]). `?engines=` isolates a single provider; `/debug` adds engine diagnostics. **3. `check.searx.space`** — the hosted official checker probes every engine of your instance with a **headless Firefox** (catches Cloudflare/CAPTCHA blocks a -plain HTTP probe misses) and posts results on searx.space. Add your instance via +plain HTTP probe misses) and posts results on searx.space. Add your instance a GitHub issue in `searxng/searx-instances`, then allow `check.searx.space` IPs to reach the SearXNG port. @@ -417,10 +417,10 @@ misleading). already replaced — checked 2026-10-18; see [[SearXNG]]). It is also immune to the shared-VPN-exit-IP blocking this page is about: it authenticates, so reputation of the exit IP is irrelevant. -5. **Verify first** — `/json` + `?engines=google` and the searx-checker to see +5. **Verify first** — `/json` + `?engines=google` and the searx-checker to exactly which engines are down and why. Confirm the running SearXNG version (check whether `parsers`/`rate_limit` exist in the pinned build). -6. **Use conservative request tuning** — Dankclaw now allows 5 seconds per +6. **Use conservative request tuning** — Dankclaw now allows 5 seconds upstream request (`shared/searxng/settings.yml:26`), caps it at 15 seconds (`:27`), and retries once (`:32`). This addresses modest VPN latency without multiplying 429/CAPTCHA traffic. Adjust only diff --git a/serverAppConfigs-docs/Dankclaw/SearXNG.md b/serverAppConfigs-docs/Dankclaw/SearXNG.md index b03662a..cb26302 100644 --- a/serverAppConfigs-docs/Dankclaw/SearXNG.md +++ b/serverAppConfigs-docs/Dankclaw/SearXNG.md @@ -108,7 +108,7 @@ engines: disabled: true ``` -The block above is the repo file (`shared/searxng/settings.yml`, 21 engine stanzas); +The block above is the repo file (`shared/searxng/settings.yml`, 21 engine stanzas) the deployed render at `/opt/searxng/config/settings.yml` differs only in the baked `secret_key`, `base_url` and `KAGI_API_KEY` values, and the running engine set is censused below. @@ -154,7 +154,7 @@ what keeps the `trusted_proxies` entry meaningful. ## Unblocking Google & Search Providers -The current request policy allows a modest amount of VPN latency (`5.0s` per +The current request policy allows a modest amount of VPN latency (`5.0s` request, `15.0s` maximum — `shared/searxng/settings.yml:26-27`) while limiting retries to one (`:32`). This can recover slow providers, but it cannot fix an upstream CAPTCHA or 429 caused by the shared @@ -169,10 +169,10 @@ upstream request volume. SearXNG **suspends** offending engines — which is wha up as "Google went unavailable". The default suspension is shorter than the folklore: see the table in [[SearXNG Providers]]. This is upstream, not local. -Fixes (cheapest first): tune the `outgoing` block (`request_timeout`, +Fixes (cheapest first): tune the `outgoing` block (`request_timeout` `retry_on_http_error`, `suspended_times`), duplicate the `google` engine under a fresh `name` so a CAPTCHA ban on one leaves the other returning results, and — -if Google still re-blocks — route the Google engines at a clean-IP proxy via +if Google still re-blocks — route the Google engines at a clean-IP proxy per-engine `proxies` in `settings.yml`. See [[SearXNG Providers]] for the full root cause, the suspend-model table, the `engines:` config, and the `/json` verification recipe. @@ -212,17 +212,16 @@ root cause, the suspend-model table, the `engines:` config, and the (`searxng.container:21`), and that is one of the variables SearXNG substitutes at load, so both paths carry the same LAN-IP value. Flagged, not fixed here: it is a config change, out of scope for docs. -- Keep the valkey cache container on the network SearXNG can actually reach — see +- Keep the valkey cache container on the network SearXNG can actually reach — [[Valkey]] Gotchas: SearXNG itself has no network of its own (it borrows Gluetun's netns), so the network that matters is the one Gluetun and `searxng-valkey` share. - ## Kagi (paid API engine) Added 2026-09-26 (user-ordered) and still enabled: the running build ships a native -`kagi` engine module (`searx/engines/kagi.py` at the pinned commit `12f8b6515`), +`kagi` engine module (`searx/engines/kagi.py` at the pinned commit `12f8b6515`) which sets `Authorization: Bearer ` (`kagi.py:103`) and issues -`POST https://kagi.com/api/v1/search` (`base_url` + `/api/v1/search`, `kagi.py:76,104,131`), +`POST https://kagi.com/api/v1/search` (`base_url` + `/api/v1/search`, `kagi.py:76,104,131`) and declares `require_api_key: True` (`kagi.py:63`). - **Enabled**: `weight: 2` alongside bing/duckduckgo, categories @@ -231,10 +230,10 @@ and declares `require_api_key: True` (`kagi.py:63`). `enabled: true`, categories `general, web` (2026-10-18). - **Cost control**: `paging: false` — exactly one API credit per aggregated query; `timeout: 8.0` so a slow paid query is not wasted (scrapers use 5.0) - (`shared/searxng/settings.yml:74-77`; upstream paging default is `True`, + (`shared/searxng/settings.yml:74-77`; upstream paging default is `True` `kagi.py:67`, so the `false` here is a real override). -- **Key handling**: the repo file keeps `api_key: "${KAGI_API_KEY}"`; - `deploy-remote.sh` bakes the real key into the live settings.yml via +- **Key handling**: the repo file keeps `api_key: "${KAGI_API_KEY}"` + `deploy-remote.sh` bakes the real key into the live settings.yml `scripts/searxng-render-settings.py`, reading `/opt/searxng/.env` (owner `dank`, mode 0600 — `ls -l /opt/searxng/.env` 2026-10-18; its key names are the WireGuard set plus `SEARXNG_SECRET`, `SEARXNG_BASE_URL`, `KAGI_API_KEY`). @@ -262,15 +261,15 @@ and declares `require_api_key: True` (`kagi.py:63`). - 262 engines in the build, **75 enabled**, categories of the enabled set led by `other` (12), `it` (11), `general` (9), `videos` (8), `stock images` (7). -- Enabled and matching the policy page: `bing`, `duckduckgo`, `wikipedia`, `kagi`, - `github`, `mdn`, `stackoverflow`, `askubuntu`, `superuser`, `docker hub`, `pypi`, - `arxiv`, `pubmed`, `semantic scholar`, `reuters`, `bing news`, `duckduckgo news`, - `wikinews`, `bing images`, `duckduckgo images/videos/news`, `wikicommons.*`, +- Enabled and matching the policy page: `bing`, `duckduckgo`, `wikipedia`, `kagi` + `github`, `mdn`, `stackoverflow`, `askubuntu`, `superuser`, `docker hub`, `pypi` + `arxiv`, `pubmed`, `semantic scholar`, `reuters`, `bing news`, `duckduckgo news` + `wikinews`, `bing images`, `duckduckgo images/videos/news`, `wikicommons.*` `youtube`, `vimeo`, `dailymotion`, `bing videos`. -- Disabled, as configured: `google`, `google cse`, `google news`, `google scholar`, +- Disabled, as configured: `google`, `google cse`, `google news`, `google scholar` `brave`, `brave.images`, `brave.news`, `brave.videos`, `qwant`. - The three `startpage` stanzas in `shared/searxng/settings.yml:113-124` target - engines that **no longer exist in this build** — `/config` lists no `startpage`, + engines that **no longer exist in this build** — `/config` lists no `startpage` `startpage images` or `startpage news`, so those overrides are no-ops kept as intent. *(labelled 2026-10-18)* diff --git a/serverAppConfigs-docs/Dankclaw/Swap Configuration.md b/serverAppConfigs-docs/Dankclaw/Swap Configuration.md index eab3eae..ba11142 100644 --- a/serverAppConfigs-docs/Dankclaw/Swap Configuration.md +++ b/serverAppConfigs-docs/Dankclaw/Swap Configuration.md @@ -144,8 +144,8 @@ frank-dragoon `health=200` throughout. The previous 8 GiB `/swap.img` (dated 2026-07-21) was staged, not deleted, at `dankclaw:~/.AGENTTRASH/20260912-swapfile-8g/swap.img` (8.1 GiB). **That staging directory no longer exists** — `ls /home/dank/.AGENTTRASH` on 2026-10-18 lists only -`20261005-frank-small-ubatch-ab`, `20261006-frank-large-tuning`, -`20261007-openwebui-modelpicker`, `20261010-fleet-rename`, `20261017-bonsai-scratch`, +`20261005-frank-small-ubatch-ab`, `20261006-frank-large-tuning` +`20261007-openwebui-modelpicker`, `20261010-fleet-rename`, `20261017-bonsai-scratch` `20261017-prism-build`, `20261017-v2abl-trial`, and there is no `/root/.AGENTTRASH`. The old swapfile has therefore been purged; nothing on the host still holds it. diff --git a/serverAppConfigs-docs/Dankclaw/Voice Packs.md b/serverAppConfigs-docs/Dankclaw/Voice Packs.md index aebeafa..7730a7d 100644 --- a/serverAppConfigs-docs/Dankclaw/Voice Packs.md +++ b/serverAppConfigs-docs/Dankclaw/Voice Packs.md @@ -23,7 +23,7 @@ voice is how a wrong-voice endpoint stays wrong for weeks. state that stays out of the repo, mounted read-only at `/voices` by the façade unit — **Builder:** `scripts/build-voice-pack.sh` with tunables in `scripts/voice-packs.env` — **Seam:** `tests/voice_pack_unit.bats` — **Engine:** -arm A `qwentts.cpp`, decided in [[Local TTS and Voice Cloning]]. +arm A `qwentts.cpp`, decided in . Packs are **world-readable on purpose** (`drwxr-xr-x`, owner `dank`): the façade container runs as uid 10000 and reads the mount from outside its own uid map. @@ -137,7 +137,7 @@ invariant to every crop in the tracked pack list. 1. Pick the source file and hash it: `sha256sum /path/to/sample.wav`. A crop offset is a claim about one exact file, so the digest goes in the env with it. 2. Find a speech-dense stretch **past 10 s**. The trial's recipe, applied once: - `ffmpeg -ss 60 -t 600 -i SRC -af silencedetect=noise=-35dB:d=0.6 -f null -`, + `ffmpeg -ss 60 -t 600 -i SRC -af silencedetect=noise=-35dB:d=0.6 -f null -` build the runs of non-silence, keep runs ≥ 18 s, take the longest (earliest on ties), and cap at 25 s. Aim for 20–25 s: shorter than 19.84 s and the `.rvq` falls under the floor. @@ -188,15 +188,15 @@ loud, not a bug. Work through it in order: 1. Re-read the tool's own numbers rather than trusting this page: `qwen-codec` prints `SpeakerEncoder] Loaded: enc_dim=…` and `Pipeline] Ready: hop … , … codebooks @ … Hz` on every run, and the pinned tree's - `tools/qwen-codec.cpp` documents the packing. `enc_dim × 4` is the `.spk`; + `tools/qwen-codec.cpp` documents the packing. `enc_dim × 4` is the `.spk` `codebooks × code_bits / 8` bytes per frame × frames is the `.rvq`. 2. Decide whether the new number is intended. A changed `enc_dim` means the old `.spk` files are unreadable by the new engine, so **every** pack must be rebuilt, not just the failing one. 3. Update the four arithmetic lines in `scripts/voice-packs.env` - (`VOICE_PACK_SPK_BYTES`, `VOICE_PACK_RVQ_MIN_BYTES`, `VOICE_PACK_RVQ_MAX_BYTES`, + (`VOICE_PACK_SPK_BYTES`, `VOICE_PACK_RVQ_MIN_BYTES`, `VOICE_PACK_RVQ_MAX_BYTES` and `VOICE_PACK_RVQ_BYTES_PER_FRAME` if the codebook packing changed) **and** - `VOICE_PACK_ENGINE_COMMIT`, `VOICE_PACK_TOOL`, `VOICE_PACK_CODEC_MODEL`, + `VOICE_PACK_ENGINE_COMMIT`, `VOICE_PACK_TOOL`, `VOICE_PACK_CODEC_MODEL` `VOICE_PACK_TALKER_MODEL` to the new build. An env diff that moves the sizes without moving the commit is the mistake to avoid. 4. Rebuild all three packs with `--force` (the retired directories keep the old @@ -226,7 +226,7 @@ loud, not a bug. Work through it in order: narrator1 and narrator2). The builder's first real run is therefore a migration, and it should be a `--force` run per voice so the old directories are retired rather than left half-updated. -- The three source files (`/home/blank/audiosample.mp3`, +- The three source files (`/home/blank/audiosample.mp3` `audiosample2.m4b`, `audiosample3.m4b`, digests pinned in `scripts/voice-packs.env`) live on the workstation, not on Dankclaw; the builder reads them from `$VOICE_PACK_SOURCE_DIR=/home/dank/voice-sources`, so diff --git a/serverAppConfigs-docs/Dankserver/Calibre.md b/serverAppConfigs-docs/Dankserver/Calibre.md index fed101c..84e2a15 100644 --- a/serverAppConfigs-docs/Dankserver/Calibre.md +++ b/serverAppConfigs-docs/Dankserver/Calibre.md @@ -80,7 +80,7 @@ image's own nginx (the desktop), so the content server runs on internal 8090. **Auth model (calibre 9.x):** users live in a SQLite userdb whose `pw` column is **plaintext** — the file is mode 600. The single admin account is bootstrapped once by the deploy handler -(`deploy_calibre_server` in `scripts/deploy-remote.sh`) via +(`deploy_calibre_server` in `scripts/deploy-remote.sh`) `calibre-server --manage-users -- add`, the password piped on stdin with `printf` (a trailing newline from `echo` lands in the stored password and breaks every login). The desktop's basic auth (`CUSTOM_USER`/`CUSTOM_PASSWORD`) diff --git a/serverAppConfigs-docs/Dankserver/Copyparty.md b/serverAppConfigs-docs/Dankserver/Copyparty.md index 87fa51c..f828c77 100644 --- a/serverAppConfigs-docs/Dankserver/Copyparty.md +++ b/serverAppConfigs-docs/Dankserver/Copyparty.md @@ -83,7 +83,7 @@ The `iv` edition is no longer available for arm32/armv6. Use `dj` for maximum co ## Volumes -Live deployment: the whole LUKS array is served at URL `/w` via +Live deployment: the whole LUKS array is served at URL `/w` `Volume=/mnt/entropy:/w:ro` in the quadlet unit ([[Entropy Volume]]) — read-only **at the mount level**, so no containerized consumer can write into the array through this path regardless of account permissions. The ac edition @@ -111,7 +111,7 @@ convention; see [[#Command-line interface (ac edition)]] above. ``` `${COPAY_ACCOUNT}`/`${COPAY_PASSWORD}` are **never** written into the unit: -they live in the host-local `/home/dank/Containers/copyparty/env` (0600), +they live in the host-local `/home/dank/Containers/copyparty/env` (0600) loaded via `[Service] EnvironmentFile=` so systemd expands them into `Exec=-a ${COPAY_ACCOUNT}:${COPAY_PASSWORD}` at start. The password still lands in the container argv — visible to anyone who can run @@ -178,6 +178,6 @@ Upgrade to v1.20.19+ to cover all known advisories. The dirkey fix (v1.20.17) ma ## Citations - GitHub Security Advisories, `9001/copyparty` — - https://github.com/9001/copyparty/security/advisories (accessed 2026-10-19); + https://github.com/9001/copyparty/security/advisories (accessed 2026-10-19) each row's patched version read from `first_patched_version` in the `https://api.github.com/advisories/` response. diff --git a/serverAppConfigs-docs/Dankserver/Dankserver Stack.md b/serverAppConfigs-docs/Dankserver/Dankserver Stack.md index 31f3972..370c2d5 100644 --- a/serverAppConfigs-docs/Dankserver/Dankserver Stack.md +++ b/serverAppConfigs-docs/Dankserver/Dankserver Stack.md @@ -54,5 +54,5 @@ The following services ran on Dankserver and are now retired from this stack: ## See Also -- [[index]] — repository index +- repository index - [[Network Architecture]] — routing and proxy diff --git a/serverAppConfigs-docs/Dankserver/Jellyfin.md b/serverAppConfigs-docs/Dankserver/Jellyfin.md index 1af373a..be05ddc 100644 --- a/serverAppConfigs-docs/Dankserver/Jellyfin.md +++ b/serverAppConfigs-docs/Dankserver/Jellyfin.md @@ -14,7 +14,7 @@ serving music, movies, and shows over LAN. It replaced the earlier research brief; it runs alongside [[Plex]], which remains the primary server for some clients. -**Host:** Dankserver (192.168.0.235) — **Access:** `http://192.168.0.235:8096`, +**Host:** Dankserver (192.168.0.235) — **Access:** `http://192.168.0.235:8096` public vhost `jelly.franksdank.cloud` behind the [[Anubis]] guard chain → [[Authentik]] outpost (post-auth route proxies `localhost:8096`). @@ -67,7 +67,7 @@ user to verify the password. - LDAP provider `Jellyfin LDAP` (Base DN `dc=franksdank,dc=cloud`, bind/search mode `direct`) + application `Jellyfin`. - LDAP outpost `Jellyfin LDAP Outpost` → container `authentik-outpost-ldap` - (quadlet `dankclaw/authentik/quadlet/authentik-outpost-ldap.container`), + (quadlet `dankclaw/authentik/quadlet/authentik-outpost-ldap.container`) publishing `389`/`636` on the Dankclaw host for cross-host binds. - Service account `svc-jellyfin-ldap` (DN `cn=svc-jellyfin-ldap,ou=users,dc=franksdank,dc=cloud`). @@ -80,7 +80,7 @@ user to verify the password. **Jellyfin side** (plugin config, set via admin → Plugins → LDAP-Auth): - Server `192.168.0.143:389` (plain LDAP over the trusted LAN), Base DN `dc=franksdank,dc=cloud`, bind user = the service-account DN, search - attributes `uid,cn,mail,displayName`, uid attr `uid`, username attr `cn`, + attributes `uid,cn,mail,displayName`, uid attr `uid`, username attr `cn` `CreateUsersFromLdap=true`, all folders enabled. **Gotcha — the bind credential is a PASSWORD, not a token.** The LDAP bind is diff --git a/serverAppConfigs-docs/Dankserver/Plex.md b/serverAppConfigs-docs/Dankserver/Plex.md index 99cbac4..682dde1 100644 --- a/serverAppConfigs-docs/Dankserver/Plex.md +++ b/serverAppConfigs-docs/Dankserver/Plex.md @@ -60,7 +60,7 @@ root runs as the `dank` user, and the deployer adopts the data root with ## Integration points - **Caddy:** Plex is exempt from the Anubis bot challenge - (plain-HTTP legacy clients cannot solve JS challenges) — see + (plain-HTTP legacy clients cannot solve JS challenges) — [[Anubis]] "Exempt by construction". - **Tautulli** ([[Tautulli]]) is the companion: it polls PMS on the appnet for statistics and notifications and makes no sense without Plex. diff --git a/serverAppConfigs-docs/Dankserver/RomM.md b/serverAppConfigs-docs/Dankserver/RomM.md index dec123e..9999074 100644 --- a/serverAppConfigs-docs/Dankserver/RomM.md +++ b/serverAppConfigs-docs/Dankserver/RomM.md @@ -32,7 +32,7 @@ Anubis → Authentik chain ([[Caddy]] routes it post-auth to ## Architecture Current RomM releases require MariaDB plus Redis (or Valkey) for the task -queue; this deployment runs **redis** (`REDIS_HOST=redis` in the live unit), +queue; this deployment runs **redis** (`REDIS_HOST=redis` in the live unit) not valkey. ```mermaid @@ -114,7 +114,7 @@ scan: ## Deployment (Quadlet) Since 2026-09-01 the trio runs as repo-tracked quadlet units, deployed with -`./scripts/deploy.sh dankserver --service romm` (any of `romm`, +`./scripts/deploy.sh dankserver --service romm` (any of `romm` `romm-mariadb`, `romm-redis` triggers the full ordered cutover): - Units: `dankserver/romm/quadlet/{romm,romm-mariadb,romm-redis}.container` diff --git a/serverAppConfigs-docs/Desktop/Noctalia and Niri.md b/serverAppConfigs-docs/Desktop/Noctalia and Niri.md index 097c5c3..62e6538 100644 --- a/serverAppConfigs-docs/Desktop/Noctalia and Niri.md +++ b/serverAppConfigs-docs/Desktop/Noctalia and Niri.md @@ -45,14 +45,14 @@ Verified: `msg status` returns live JSON state; unknown panel ids return an error listing all valid panel ids (clipboard, control-center, launcher, polkit, session, setup-wizard, test, tray-drawer, wallpaper). -Verified v5 msg verb grammar (keybinds-safe): `panel-toggle `, -`session lock|suspend|lock-and-suspend|logout|reboot|shutdown`, -`volume-up [step]`, `volume-down`, `volume-mute`, `mic-mute`, +Verified v5 msg verb grammar (keybinds-safe): `panel-toggle ` +`session lock|suspend|lock-and-suspend|logout|reboot|shutdown` +`volume-up [step]`, `volume-down`, `volume-mute`, `mic-mute` `media next|previous|playPause` (graceful no-op, rc 0, with no active -player), `brightness-up [target] [step]`, `brightness-down`, -`settings-open [context]`, `settings-toggle [context]`, `config-reload`, -`wallpaper-get [connector]`, `wallpaper-set `, `wallpaper-next`, -`color-scheme-get|set`, `theme-mode-get`, `templates-apply`, +player), `brightness-up [target] [step]`, `brightness-down` +`settings-open [context]`, `settings-toggle [context]`, `config-reload` +`wallpaper-get [connector]`, `wallpaper-set `, `wallpaper-next` +`color-scheme-get|set`, `theme-mode-get`, `templates-apply` `screenshot-fullscreen`. **Wallpaper directory semantics:** empty/unset `[wallpaper] directory` @@ -81,9 +81,9 @@ Custom palettes live at `~/.config/noctalia/palettes/.json` synced. Missing/invalid file → silent fallback to the builtin palette. Palette JSON shape = the community file's shape: `dark`/`light` maps of -`mPrimary`, `mOnPrimary`, `mSecondary`, `mTertiary`, `mError`, `mSurface`, +`mPrimary`, `mOnPrimary`, `mSecondary`, `mTertiary`, `mError`, `mSurface` `mSurfaceVariant`, `mOutline`, `mShadow`, `mHover` plus a `terminal` block -(`foreground`, `background`, `normal.{black..white}`, `bright.{…}`, +(`foreground`, `background`, `normal.{black..white}`, `bright.{…}` `cursor`, `cursorText`, `selectionFg`, `selectionBg`). NOTE: `noctalia theme -o` emits M3 snake_case tokens (`on_primary_container`, …) — that is the *generator* format, not this one. Copy an existing community JSON and mutate @@ -114,7 +114,7 @@ Procedure: `noctalia msg color-scheme-set custom "Gruber Red"` → `ok`; confirm with `color-scheme-get`. 5. Re-render every configured template: `noctalia msg templates-apply`. - Regenerates `~/.config/ghostty/themes/noctalia`, + Regenerates `~/.config/ghostty/themes/noctalia` `~/.config/gtk-{3,4}.0/noctalia.css`, kcolorscheme, foot, wezterm, starship, etc. Symlinked-into-dotfiles paths show up as modified files worth committing. @@ -216,7 +216,7 @@ Verified on blankpad, niri 26.04 (two PM1601 externals + eDP-1 laptop panel). scale, transform, logical pos/size) and `niri --version`. Output **names are connector names** (`eDP-1`, `DP-2`, `HDMI-A-1`), not EDID model strings — the EDID strings printed before them are decoration. On blankpad the live -config splits via `include "./cfg/*.kdl"` from `~/.config/niri/config.kdl`, +config splits via `include "./cfg/*.kdl"` from `~/.config/niri/config.kdl` and `cfg/display.kdl` is a **symlink into the dotfiles repo** — edit the repo path (that is what gets committed); check with `ls -ld`. diff --git a/serverAppConfigs-docs/FrankBook-Pro/ARR-Stack.md b/serverAppConfigs-docs/FrankBook-Pro/ARR-Stack.md index 4ffaf42..be71434 100644 --- a/serverAppConfigs-docs/FrankBook-Pro/ARR-Stack.md +++ b/serverAppConfigs-docs/FrankBook-Pro/ARR-Stack.md @@ -38,7 +38,7 @@ live call at a time instead of guessing payload shapes: - `'Base Url' must not be empty` when keys were camelCase → **PascalCase is required** (`BaseUrl`, `ApiKey`) — schema keys, not the JSON wire keys. - `DefaultMetadataProfileId must be > 0` → fetch real ids from - `GET /api/v1/metadataprofile` (rows like `(1,'Standard')`); + `GET /api/v1/metadataprofile` (rows like `(1,'Standard')`) `GET /api/v1/qualitydefinition` returns the quality CATALOG — ids ≠ profiles. - `targetType null` → configContract not registered in that build (this @@ -87,7 +87,7 @@ settings values live in `prowlarr.db` → `Applications.Settings` JSON (the pipeline hides a miss and you conclude "empty field" when the grep itself failed. 4. Decision framing: when posture = gated-with-zero-users, present it as a - decision menu (None behind the front-door wall vs External/authentikarr), + decision menu (None behind the front-door wall vs External/authentikarr) not a silent change — first-account-wins means whoever logs in first owns the instance. diff --git a/serverAppConfigs-docs/FrankBook-Pro/prowlarr.md b/serverAppConfigs-docs/FrankBook-Pro/prowlarr.md index b204f8a..d58bf40 100644 --- a/serverAppConfigs-docs/FrankBook-Pro/prowlarr.md +++ b/serverAppConfigs-docs/FrankBook-Pro/prowlarr.md @@ -70,7 +70,7 @@ http://prowlarr:9696/{indexerId}/download?link=...&file=... (torrent handoff) The global API key is in `config.xml` (``); per-application keys are only used by Prowlarr's own sync calls, NOT by the feeds. The arr download clients authenticate to qBittorrent with the permanent WebUI password stored -in `/home/dank/.qbpass` (mode 600, owner `dank`) on Dankserver — see +in `/home/dank/.qbpass` (mode 600, owner `dank`) on Dankserver — [[qBittorrent]] for why that password must be pinned in the conf file. **Readarr caveat:** Readarr is abandoned upstream; its qBittorrent client @@ -97,13 +97,13 @@ successor; the indexer row itself is healthy. - Categories in the indexer row must be a subset of what the feed's `caps` advertises; extras log "does not support the following categories" and some feeds reject them. -- **A freshly created indexer row defaults to all-off.** If `enableRss`, +- **A freshly created indexer row defaults to all-off.** If `enableRss` `enableAutomaticSearch`, or `enableInteractiveSearch` aren't set, the app logs `Searching indexers for [X]. 0 active indexers` (Lidarr, verified). Search proves TV + music chains 2026-08-30: Sonarr returned 76 live releases for an anime term, Lidarr 18 for YOASOBI – THE BOOK, all via the Prowlarr-Nyaa feed. -- The v1 API collection is `/api/v1/applications` (not `/api/v1/app`); +- The v1 API collection is `/api/v1/applications` (not `/api/v1/app`) GET responses carry `fields: []` — real settings values live in `prowlarr.db` → `Applications.Settings` JSON (read-only inspection). - Instance name/`InstanceName=Prowlarr` in `config.xml` does NOT prefix the diff --git a/serverAppConfigs-docs/FrankBook-Pro/qbittorrent.md b/serverAppConfigs-docs/FrankBook-Pro/qbittorrent.md index 65a8446..221637b 100644 --- a/serverAppConfigs-docs/FrankBook-Pro/qbittorrent.md +++ b/serverAppConfigs-docs/FrankBook-Pro/qbittorrent.md @@ -13,7 +13,7 @@ qBittorrent is the torrent download client for the *arr stack. The stack runs on not on Dankserver. It receives download requests from Sonarr, Radarr, Lidarr, and Readarr via category tagging. -**Host:** FrankBook-Pro (podman machine) — **Access:** WebUI on port `8080`, +**Host:** FrankBook-Pro (podman machine) — **Access:** WebUI on port `8080` published by the [[Gluetun]] unit because qBittorrent shares that container's network namespace (a netns-sharer publishes no ports of its own). @@ -121,7 +121,7 @@ Mac shell export `PATH="/usr/local/bin:/opt/homebrew/bin:$PATH"` for `podman machine ssh`. **Access note (2026-09-21):** the Mac's login user is **dank**@192.168.0.112 -(NOT core — `core` is the podman-machine VM user, reached only via +(NOT core — `core` is the podman-machine VM user, reached only `podman machine ssh`). blankpad has key auth for `dank@` (bootstrapped 2026-09-21) and a `frankbook-pro` alias in `~/.ssh/config`. macOS sshd has a low MaxAuthTries: when password-bootstrapping use `-o diff --git a/serverAppConfigs-docs/Infrastructure/Anubis.md b/serverAppConfigs-docs/Infrastructure/Anubis.md index cab9647..71d01c1 100644 --- a/serverAppConfigs-docs/Infrastructure/Anubis.md +++ b/serverAppConfigs-docs/Infrastructure/Anubis.md @@ -93,7 +93,7 @@ and `hermesadmin.` (app-enforced passwords, no public DNS for the first, (`dankclaw/Caddyfile:241-251`), and the FrankBook-Pro *arr/qBittorrent vhosts (`dankclaw/Caddyfile:260-352`). There is also no vhost at all for ComfyUI, so it is exempt by absence — and therefore unauthenticated on the LAN at -`comfyui.franksdank.cloud:8188`. The guarded set is exactly `www.`, `search.`, +`comfyui.franksdank.cloud:8188`. The guarded set is exactly `www.`, `search.` `grafana.`, `prometheus.`, `loki.`, `seerr.`, `marreta.`, `romm.`, `files.` (`dankclaw/Caddyfile:283-423`). @@ -140,7 +140,7 @@ it is exempt by absence — and therefore unauthenticated on the LAN at instead of allowed, the browser's fetch throws and it renders its own error page (iOS/macOS browsers phrase it as "the request failed and the interceptors did not return an alternative response") instead of the - login flow. Verify with a JS-capable client end-to-end — mint via + login flow. Verify with a JS-capable client end-to-end — mint `make-challenge`, solve, submit with browser-exact query params — before declaring a challenge chain healthy. 7. **Single-file bind mount + git pull = stale policy.** The policy is @@ -158,7 +158,7 @@ it is exempt by absence — and therefore unauthenticated on the LAN at answers any request lacking `X-Real-Ip` with a 500 whose log line reads `[misconfiguration] X-Real-Ip header is not set`. The real chain always sets it (every guarded Caddy listener carries `header_up X-Real-Ip - {remote_host}` — the bats seam enforces the line), so users never see + {remote_host}` — the bats seam enforces the line), so users never this. But any direct probe of `:8923` (scratch container, curl, a new reverse-proxy hop) MUST set the header or it looks like a policy failure. Scratch-verify with `-H "X-Real-Ip: 192.168.0.20"`. diff --git a/serverAppConfigs-docs/Infrastructure/Authentik Outpost Reference.md b/serverAppConfigs-docs/Infrastructure/Authentik Outpost Reference.md index 6139884..a3e1f4e 100644 --- a/serverAppConfigs-docs/Infrastructure/Authentik Outpost Reference.md +++ b/serverAppConfigs-docs/Infrastructure/Authentik Outpost Reference.md @@ -23,8 +23,8 @@ read-only `SELECT`s, 2026-10-18). Nine proxy providers, all attached to outpost `7e39655b-58d3-4fe6-a80a-513f31051217` ("authentik Embedded Outpost", 9 members in `authentik_outposts_outpost_providers`), all with `internal_host = http://caddy:8017`, all carrying the **same three flow -UUIDs** (`default-authentication-flow`, -`default-provider-authorization-implicit-consent`, +UUIDs** (`default-authentication-flow` +`default-provider-authorization-implicit-consent` `default-provider-invalidation-flow`), and each backed by an Application (the `slug` column below): @@ -67,7 +67,7 @@ Two provisioning rules, each verified live by an outage: reload** — including hosts whose providers were fine (verified 2026-09-02 when three providers provisioned without any flows took down every guarded vhost after a server restart). Backfill all three flows (copy from pk 1: - `default-authentication-flow`, + `default-authentication-flow` `default-provider-authorization-implicit-consent` (the implicit-consent authorization flow, `default-invalidation-flow`) and the retry loop recovers on its next 3-second cycle. The silent killer: while the outpost @@ -94,7 +94,7 @@ implementation diverges from upstream in ways that bit us; all verified live: contains `"authorization_code"` (enum value with underscore), the token endpoint rejects every exchange with `invalid_request`. - **Token endpoint is form-encoded + Basic-auth only.** A JSON body makes - `parse_token_request` see no `grant_type` → `unsupported_grant_type`; + `parse_token_request` see no `grant_type` → `unsupported_grant_type` POST-body client secrets are not accepted — authenticate the confidential client with `Authorization: Basic base64(client_id:client_secret)`. - **Provider resolution is by `client_id` query param** on the global @@ -107,7 +107,7 @@ implementation diverges from upstream in ways that bit us; all verified live: becomes the *expression source* instead of the context; every call raises a swallowed `PropertyMappingExpressionException` and the claim is skipped. Scope-grant filtering has the same class of bug: `ScopeMapping` has no - provider FK (linkage lives in the `ProviderPropertyMapping` through model), + provider FK (linkage lives in the `ProviderPropertyMapping` through model) so `filter(provider=...)` always yields an empty queryset and tokens asked to carry scopes get zero. Net effect: userinfo/introspection answers with `{sub}` only — email/groups/entitlements claims cannot be delivered by this @@ -156,7 +156,7 @@ Hard-won rules (the first is now contested by the live database): and retest. - `ak shell` import path for the provider model in 2026.8: `from authentik.providers.ldap.models import LDAPProvider`. - *(unverified 2026-10-18: importing it outside a configured Django shell fails + - (unverified 2026-10-18: importing it outside a configured Django shell fails with `ImproperlyConfigured`, and no `ak shell` was driven against the live auth service for this check)* @@ -188,13 +188,13 @@ is a `password` field on the `default-user-settings` prompt Two ORM gotchas: `Flow.stages.add()` fails on this through model (`Can't bulk create a multi-table inherited model` — `FlowStageBinding` is -MTI), so create the `FlowStageBinding` row directly with an explicit `order`; +MTI), so create the `FlowStageBinding` row directly with an explicit `order` and a prompt field is a `Prompt` row linked through `PromptStageField` (`prompt_stage` + `prompt`; the order comes from `Prompt.order`). **A blank password must not set an empty password.** `set_password("")` would make the account log in with no password, so a validation policy -(`default-user-settings-no-empty-password`, bound to the prompt via +(`default-user-settings-no-empty-password`, bound to the prompt `PromptStageValidationPolicy`) denies the form when `prompt_data["password"]` is `""` and allows it when set. Verified with the `PolicyEvaluator` (empty → deny, set → allow, absent → allow) and end-to-end by driving @@ -213,7 +213,7 @@ three `*_flow_id` columns; the proxy table is `authentik_providers_proxy_proxyprovider` (doubled `proxy`) and its PK column really is **`oauth2provider_ptr_id`** (proxy providers inherit the OAuth2 table in 2026.8), the LDAP table is -`authentik_providers_ldap_ldapprovider` with PK `provider_ptr_id`; +`authentik_providers_ldap_ldapprovider` with PK `provider_ptr_id` `authentik_core_application` attaches to a provider through `provider_id` (its PK is `policybindingmodel_ptr_id`, not `id`), and `authentik_outposts_outpost` is keyed by `uuid`. So `Provider.objects.filter(type=…)` raising `FieldError` diff --git a/serverAppConfigs-docs/Infrastructure/Authentik.md b/serverAppConfigs-docs/Infrastructure/Authentik.md index 2d1578f..d43def8 100644 --- a/serverAppConfigs-docs/Infrastructure/Authentik.md +++ b/serverAppConfigs-docs/Infrastructure/Authentik.md @@ -15,20 +15,20 @@ application-level authentication with its proxy outpost. Not wrapped: Gitea, OpenWebUI, Jellyfin (LDAP outpost), the *arr stack, Pi-hole (basicauth edge) — they handle their own session. Grafana is the exception that is both: it is a guarded vhost (`grafana-proxy`, pk 8, locked to Authentik SSO) *and* an OAuth2 -relying party (`grafana`, pk 9) (`dankclaw/Caddyfile:122-132`, +relying party (`grafana`, pk 9) (`dankclaw/Caddyfile:122-132` `dankclaw/Caddyfile:374-377`; live provider table in [[Authentik Outpost Reference]]). -- **Version:** `2026.8.3` (pinned in `dankclaw/authentik/quadlet/*.container`; - the outpost's boot log reads `"Booting authentik" … "version": "2026.8.3"`, +- **Version:** `2026.8.3` (pinned in `dankclaw/authentik/quadlet/*.container` + the outpost's boot log reads `"Booting authentik" … "version": "2026.8.3"` 2026-10-18) - **Host:** dankclaw (`192.168.0.143`) - **Name:** `auth.franksdank.cloud` -- **Stack: five quadlet units, no compose.** `authentik-postgres`, - `authentik-server`, `authentik-worker` (`deploy_authentik`, +- **Stack: five quadlet units, no compose.** `authentik-postgres` + `authentik-server`, `authentik-worker` (`deploy_authentik` `scripts/deploy-remote.sh:980-1004`) plus two outposts — `authentik-outpost` - (the Rust **proxy** outpost that actually serves `/outpost.goauthentik.io/*`, + (the Rust **proxy** outpost that actually serves `/outpost.goauthentik.io/*` `deploy_authentik_outpost`, `scripts/deploy-remote.sh:1006-1018`) and - `authentik-outpost-ldap` (`deploy_authentik_outpost_ldap`, + `authentik-outpost-ldap` (`deploy_authentik_outpost_ldap` `scripts/deploy-remote.sh:1020-1032`). All five report `generated` + `active`, `Restart=always`, `WantedBy=default.target`, with linger on (`systemctl --user`, 2026-10-18). The "3 compose services" wording is the @@ -36,7 +36,7 @@ relying party (`grafana`, pk 9) (`dankclaw/Caddyfile:122-132`, - **LDAP outpost:** serves LDAP for Jellyfin — see the section below. **Split off from this page:** [[Authentik Processes]] owns the per-container -facts for `authentik-postgres`, `authentik-server` and `authentik-worker`; +facts for `authentik-postgres`, `authentik-server` and `authentik-worker` [[Authentik Outpost Reference]] owns the provider/flow/token lookup material — the live proxy-provider inventory, the proxy-provider provisioning rules, the OAuth2 relying-party quirks, the LDAP provider objects, the user-settings flow @@ -136,7 +136,7 @@ routing table, and the Authentik proxy-provider list must be the same set of hosts. A guarded host with no provider 404s at login time; a provider with no DNS record breaks off-LAN clients. -The live proxy-provider inventory (the per-vhost `external_host` → app mapping), +The live proxy-provider inventory (the per-vhost `external_host` → app mapping) the two provisioning rules the outpost enforces, and this fork's OAuth2 relying-party quirks are reference lookup: [[Authentik Outpost Reference]]. @@ -144,7 +144,7 @@ relying-party quirks are reference lookup: [[Authentik Outpost Reference]]. The second outpost — `dankclaw/authentik/quadlet/authentik-outpost-ldap.container` → `authentik-outpost-ldap.service`, one of the five units in this stack — -serves LDAP for Jellyfin's LDAP-Auth plugin (`enabled`=generated, `active`, +serves LDAP for Jellyfin's LDAP-Auth plugin (`enabled`=generated, `active` `Restart=always`, container up, 2026-10-18). The image is pinned by its **amd64 single-platform digest** `ghcr.io/goauthentik/ldap@sha256:2e7b259eb8c37faac916946ddf48ec681ee9d3a9f724fe8521bf161973f13cf9` @@ -159,7 +159,7 @@ rule at the current `2026.8.3` pin. Resolve once with `podman manifest inspect .digest'` and pin that digest. - Ports **389** (ldap) and **636** (ldaps) publish on the LAN - (`PublishPort=389:3389`, `636:6636`, + (`PublishPort=389:3389`, `636:6636` `dankclaw/authentik/quadlet/authentik-outpost-ldap.container:30-31`; `ss -ltnp` shows `0.0.0.0:389` and `0.0.0.0:636`, 2026-10-18) and Dankserver can reach them — a bare TCP connect from Dankserver to `192.168.0.143:389` succeeds @@ -196,7 +196,7 @@ every outpost session (see the ORM playbook). core quadlets, starts postgres health-gated (`wait_container_healthy authentik-postgres 120`), then the server (300 s health wait) and the worker. The outposts are separate handlers: `--service authentik-outpost` and - `--service authentik-outpost-ldap` (`scripts/deploy-remote.sh:1006-1032`, + `--service authentik-outpost-ldap` (`scripts/deploy-remote.sh:1006-1032` dispatch at `:1579-1588`). 2. Open `https://auth.franksdank.cloud` — the initial setup wizard runs. It creates the superuser (email + password) and generates the DB schema. @@ -210,7 +210,7 @@ every outpost session (see the ORM playbook). 4. For apps with native OIDC: add an **OAuth2/OpenID Connect** provider, create the app with the correct redirect URI, and paste the `client_id` / `client_secret` into the app's SSO settings. The relying parties that exist - today are `grafana` (pk 9), `Seerr OIDC` (pk 12) and `Open WebUI` (pk 14); + today are `grafana` (pk 9), `Seerr OIDC` (pk 12) and `Open WebUI` (pk 14) **Gitea has no provider row at all** (live provider table, 2026-10-18) — it authenticates itself and stays off the chain by design (`dankclaw/Caddyfile:110-120`). @@ -243,7 +243,7 @@ every outpost session (see the ORM playbook). The embedded outpost does not re-evaluate group membership per request. Each completed authorize writes one row to `authentik_providers_proxy_proxysession` whose `session_data` JSON holds the claims (`groups` included) that get served -for that browser session until the row expires (observed TTLs 1–24 h per +for that browser session until the row expires (observed TTLs 1–24 h row). A session authorized **before** a user was added to a group keeps serving the old group list — Caddy-side RBAC header gates then 403 despite correct membership, and no authentik event is generated (the 403 happens at @@ -309,7 +309,7 @@ client → TLS edge → Anubis :8923 → outpost decision → internal 000 (connection refused) — the post-auth listener lives in the caddy container's netns; probe inside it or grep the config. Non-browser clients cannot solve the JS challenge — before flipping, audit service-to-service -traffic: ingestion/push paths must target internal addresses (`loki:3100`), +traffic: ingestion/push paths must target internal addresses (`loki:3100`) never a public hostname of a guarded service. Exempt-by-construction categories stay unguarded: the IdP itself (`auth.`), git smart-HTTP endpoints, services with their own complete auth — document the exemption @@ -336,14 +336,14 @@ EOF `sed -n '/objects imported/,$p'` to drop the noise *(unverified 2026-10-18: the exact banner text and the 229-object count — no `ak shell` was driven against the live auth service for this audit)*. -- Import paths (each wrong guess costs a failed exec): `User`, `Group`, +- Import paths (each wrong guess costs a failed exec): `User`, `Group` `Application`, `ApplicationEntitlement`, `PropertyMapping` → `authentik.core.models`; `ProxyProvider` → - `authentik.providers.proxy.models` (NOT `authentik.outposts.models`); - `LDAPProvider` → `authentik.providers.ldap.models`; + `authentik.providers.proxy.models` (NOT `authentik.outposts.models`) + `LDAPProvider` → `authentik.providers.ldap.models` `authentik.core.models.application` does NOT exist as an import path; there is no `authentik.policies.group` module in this build. - *(unverified 2026-10-18: these module paths were not re-executed — importing + - (unverified 2026-10-18: these module paths were not re-executed — importing outside a configured Django shell fails with `ImproperlyConfigured`)* - The live database's table and column names (the 2026.x renames, the MTI parent keys, the read-only `psql` audit recipe) are reference lookup: @@ -376,7 +376,7 @@ downstream — `category: user` panels from `services.json`. - `openwebui_users` / `openwebui_admins` → ApplicationEntitlements **"Open WebUI Users"** and **"Open WebUI Admins"**, both rows attached to the - `Open WebUI` application (`authentik_core_applicationentitlement`, + `Open WebUI` application (`authentik_core_applicationentitlement` 2026-10-18), consumed by the "Open WebUI roles" property mapping → `roles` claim *(unverified 2026-10-18: the group→entitlement binding and the mapping expression were not read; the mapping itself is not visible under that name @@ -392,7 +392,7 @@ downstream — artem 16, kirby 17, ry 18, nic 19, debo 20, freya 21, mitch 22, eric 23 — and every one carries the `@franksdank.cloud` address the template prescribes. Creation dates (2026-10-07 → 2026-10-15) were not re-checked - *(unverified 2026-10-18)*. Template: `username` + + - (unverified 2026-10-18)*. Template: `username` + `@franksdank.cloud` + display name + `is_active=True` + the password set via `set_password`, membership in `Non-Admin Users` + `jellyfin_users` + `openwebui_users`. `alex` was recreated from scratch on 2026-10-08 into this @@ -400,7 +400,7 @@ downstream — the row carried no MFA device, token, entitlement, or proxy session, and events survive a user delete because `Event.user` is a denormalized JSON field, not a FK). -- EXCLUDE from a user-level account: `homepage-admins`, `jellyfin_admins`, +- EXCLUDE from a user-level account: `homepage-admins`, `jellyfin_admins` `openwebui_admins`, `authentik Admins`, `authentik Read-only`. - Other user-category vhosts (SearXNG, Gitea, Calibre-Web, RomM, Copyparty, Plex, Seerr, Marreta) need no per-user group: any authenticated user @@ -429,8 +429,8 @@ More field gotchas (each cost a failed exec): `ApplicationEntitlement.applicatio can be null (guard with `getattr`); `LDAPProvider` has `search_mode`/`bind_mode`/`base_dn` and lacks `search_base`/`of_groups`/ `organization` — the backing table -(`authentik_providers_ldap_ldapprovider`) has exactly `provider_ptr_id`, -`base_dn`, `certificate_id`, `tls_server_name`, `gid_start_number`, +(`authentik_providers_ldap_ldapprovider`) has exactly `provider_ptr_id` +`base_dn`, `certificate_id`, `tls_server_name`, `gid_start_number` `uid_start_number`, `search_mode`, `bind_mode`, `mfa_support` (`information_schema.columns`, 2026-10-18); `OAuth2Provider` has no `.scopes` (scopes are linked through `ProviderPropertyMapping`, which is why @@ -484,5 +484,5 @@ podman run --rm --network container: --entrypoint /bin/busybox \ - [Jellyfin](../Dankserver/Jellyfin.md) — consumes the LDAP outpost via its LDAP-Auth plugin - [Network Architecture](../Architecture/Network%20Architecture.md) — the flat naming scheme - Research: `.docaudit/research/auth.md` — the full provider comparison - *(the file is gone from the repo and from the host checkout as of + - (the file is gone from the repo and from the host checkout as of 2026-10-18)* diff --git a/serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md b/serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md index c712434..2894f1b 100644 --- a/serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md +++ b/serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md @@ -47,7 +47,7 @@ torrent. WIREGUARD_PERSISTENT_KEEPALIVE_INTERVAL=25s The private key is hashed into the VM env file and never enters the repo. -NAT-PMP inbound forwarding is available on Proton (gluetun supports it via +NAT-PMP inbound forwarding is available on Proton (gluetun supports it `FIREWALL_INPUT_PORTS` + `NATPMP=true`) — deliberately deferred: the e2e gate passes outbound-only, and swarm peers needing an inbound port is a seeding-era concern, not a grab-only one. @@ -86,7 +86,7 @@ are unaffected. got `Unauthenticated`); the live conf was patched and `scripts/deploy-remote.sh` fixed to write the `_PBKDF2` key. Gotcha (live, 2026-09-01): after qB 5.x ran, its conf rewrite left a hash line - 4.6.7 rejects (`invalid credentials` for every login, including the arrs); + 4.6.7 rejects (`invalid credentials` for every login, including the arrs) rewriting the same `@ByteArray` line fresh (same password, 100k iterations) restored login. Any future version flip: regenerate the line. - `HostHeaderValidation=false` + `ServerDomains=*`: the socat forwarder @@ -98,7 +98,7 @@ are unaffected. path (no auto-TMM), so the default itself points at `Finished Torrents`: every grab lands there regardless of client behavior; human review moves files onward; nothing writes to the entropy volume. -- Categories seeded: `Finished Torrents`, `lidarr`, `radarr`, `readarr`, +- Categories seeded: `Finished Torrents`, `lidarr`, `radarr`, `readarr` `tv-sonarr` (all → `Finished Torrents`). - **Download clients** (gateway `10.88.0.1:8080`, user `admin`): all five test green against live qB — sonarr (`tv-sonarr`), radarr, lidarr, readarr @@ -109,12 +109,12 @@ are unaffected. as "Authentication failure" — that kills runtime auth too, not just API creates; the fix was pinning qB itself (below), after which readarr tests 200 like the rest. -- **qB pinned to 4.6.7** (`Image=lscr.io/linuxserver/qbittorrent:4.6.7`, +- **qB pinned to 4.6.7** (`Image=lscr.io/linuxserver/qbittorrent:4.6.7` not `latest`): `latest` resolved to 5.x, whose `204` login breaks Readarr's frozen client; 4.6.7 answers the legacy `200` + `"Ok."` that every client accepts. Pinned in `frankbook-pro/qbittorrent/quadlet/qbittorrent.container`. -- **Prowlarr Applications** push to sonarr/radarr/lidarr (`fullSync`, +- **Prowlarr Applications** push to sonarr/radarr/lidarr (`fullSync` gateway ports). Readarr deliberately omitted: prowlarr's indexer-push test to readarr's `/api/v1/indexer/test` 500s (EOL schema drift). Legacy `Prowlarr-*` indexer rows carried in the migrated dankserver DBs diff --git a/serverAppConfigs-docs/Infrastructure/Grafana.md b/serverAppConfigs-docs/Infrastructure/Grafana.md index a7e6519..a10bb1a 100644 --- a/serverAppConfigs-docs/Infrastructure/Grafana.md +++ b/serverAppConfigs-docs/Infrastructure/Grafana.md @@ -46,7 +46,7 @@ The Prometheus datasource is declared in git, not clicked into `grafana.db`: address Caddy proxies to, never the loopback-published port — and carries `timeInterval: 15s` so Grafana's rate windows cannot be shorter than the scrape interval (`prometheus.yml:5`). The `llm-fleet` dashboard is provisioned -alongside it by `dashboards/provider.yml` (provider `fleet`, +alongside it by `dashboards/provider.yml` (provider `fleet` `updateIntervalSeconds: 30`, `allowUiUpdates: true`). A Loki datasource is still **not** provisioned — the tree contains exactly one @@ -70,7 +70,7 @@ Grafana responds on `/api/health` with 200 and `"database":"ok"`; measured 2026- Basic client auth, form-encoded token body, and the fork's userinfo-claims bug that leaves identity at the `sub` hash only) is in [[Authentik Outpost Reference]] → "OAuth2 relying-party quirks". Consequences for Grafana: - The unit sets `ALLOW_SIGN_UP=false`, `AUTO_UPSERT=true`, `ALLOW_ASSIGN_ORG=true`, + The unit sets `ALLOW_SIGN_UP=false`, `AUTO_UPSERT=true`, `ALLOW_ASSIGN_ORG=true` `ORG_ID=1` and **no** role-attribute mapping (`grafana.container:52-55`), so users land as Viewer with `login = `; no email/groups/role claims reach Grafana until the evaluator bug is fixed — do not write role mappings @@ -79,7 +79,7 @@ Grafana responds on `/api/health` with 200 and `"database":"ok"`; measured 2026- the host without the database or an admin password. *(unverified 2026-10-18)*) - **Never point this unit's `EnvironmentFile` at the repo-root `.env`.** That file is a gluetun env file and it carries `HOME=/root`; systemd imports it - for the service, podman then looks for its configuration under `/root`, + for the service, podman then looks for its configuration under `/root` cannot read it, and the unit dies at start with `stat /root/.config: permission denied`. The unit reads `/home/dank/Containers/grafana/env`, which the deployer seeds with exactly diff --git a/serverAppConfigs-docs/Infrastructure/Homepage Status Probes.md b/serverAppConfigs-docs/Infrastructure/Homepage Status Probes.md index 83284d8..530b541 100644 --- a/serverAppConfigs-docs/Infrastructure/Homepage Status Probes.md +++ b/serverAppConfigs-docs/Infrastructure/Homepage Status Probes.md @@ -51,7 +51,7 @@ See [[Container Host Reachability]] for the netns rules. published nothing → structurally unprobeable from the host). 3. Probe from the **correct vantage**: from a bridge-net container the host is its gateway IP (`/proc/net/route`, e.g. `10.89.1.1` on - `dankclaw_default`) — *not* `host.containers.internal` (`169.254.1.2`), + `dankclaw_default`) — *not* `host.containers.internal` (`169.254.1.2`) which belongs to the default `podman` net (`10.88.0.0/16`). Probe with bash `/dev/tcp` (no `python3`/`ip` in these images); escape loop vars so they expand **inside** `podman exec bash -c`. @@ -83,7 +83,7 @@ python3 -m http.server 8899 --bind 127.0.0.1 ``` Then in the browser: load `http://127.0.0.1:8899/index.html`, **wait ~1.5 s** -(panels render after the async fetch; checking immediately returns 0 panels), +(panels render after the async fetch; checking immediately returns 0 panels) extract via `tab.evaluate` — LLM cards render `.desc-role`/`.desc-spec`/ `.desc-inputs`/`.desc-use` from the card's `details` object, other cards a plain `.desc`; `getComputedStyle(el).color` confirms tones. Screenshot for diff --git a/serverAppConfigs-docs/Infrastructure/Homepage.md b/serverAppConfigs-docs/Infrastructure/Homepage.md index 243a56b..03da983 100644 --- a/serverAppConfigs-docs/Infrastructure/Homepage.md +++ b/serverAppConfigs-docs/Infrastructure/Homepage.md @@ -22,7 +22,7 @@ The homepage is live on Dankclaw. `./scripts/deploy.sh dankclaw` copies the stat **Docs page shipped (2026-09-17):** `/docs` serves a static user documentation page (`shared/homepage/docs.html`) — access walkthrough, the running-model guide, media how-tos, and troubleshooting. Per-service `/docs//` pages -remain a separate thread. The three LLM cards (`frank-dragoon`, `frank-medium`, +remain a separate thread. The three LLM cards (`frank-dragoon`, `frank-medium` `frank-zealot`) moved from admin-only to the **user** page with live model descriptions and a `docs` link to `/docs#models`. @@ -59,7 +59,7 @@ without which the picker showed "No results found" for every family account "equivalent to Claude Sonnet 5 / OpenAI-Luna (Summer 2026 frontier intelligence)", frank-archon "equivalent to Claude Opus 4.8 / OpenAI-Terra (same era)", frank-zealot "equivalent to last year's frontier level -(2025)". Rendered by `descHTML()` as `.desc-equiv` (head color, italic); +(2025)". Rendered by `descHTML()` as `.desc-equiv` (head color, italic) pinned in `homepage_config.bats`. **Protoss rename + Model Information component (2026-10-08):** the model @@ -91,7 +91,7 @@ same renames, api-id mapping, supervision and compaction notes. app card now carries hex-clipped art beside its name (`services.json` `art` field, files under `img/apps/`): 19 brand marks tinted amber from the Simple Icons CDN plus 19 hand-improvised monogram SVGs (amber hex + VT323 -initials) for apps with no mark — Copyparty, Marreta, Hermes (+dashboard), +initials) for apps with no mark — Copyparty, Marreta, Hermes (+dashboard) node-exporter, Gluetun, OpenWebUI, Calibre-Web, RomM, Seerr, Loki, Alloy, Alertmanager, Valkey, Prowlarr, Readarr, Lidarr, Tautulli, ComfyUI. Model cards render in the order zealot → dragoon → archon (registry order is the @@ -143,8 +143,8 @@ per-app colors: Copyparty gold, Marreta brick, Grafana orange, Loki indigo, Neo4j network, Alloy violet, Promtail pink, Alertmanager red, Prometheus orange, Node Exporter teal, cAdvisor sky, Guake yellow, Authentik blue, Seer green radar, RomM a purple GameBoy. New per-service -fields: `art_bg` (art backdrop — OpenWebUI's dark logo needs `#f5f5f5`), -`warn` on OpenWebUI (searches ride the Seerr→SearXNG path), +fields: `art_bg` (art backdrop — OpenWebUI's dark logo needs `#f5f5f5`) +`warn` on OpenWebUI (searches ride the Seerr→SearXNG path) `model.compaction` (the amber "set this as compaction model" callout frank-zealot carries). frank-dragoon's card portrait is the SC2 unit portrait (fandom wiki `Dragoon_portrait.jpg`, webp→jpeg). The whole @@ -175,7 +175,7 @@ titles are chamfered plates with the diamond glyph inside the plate. Fonts: body is **Share Tech Mono** (legible retro terminal), headings are **Rajdhani 600** (the StarCraft UI face) — VT323 is retired; all three pages load the pair from Google Fonts. Four tiles that had -reverted to initials got pictograms: Calibre-Web an open book (orange), +reverted to initials got pictograms: Calibre-Web an open book (orange) RomM a purple GameBoy, Copyparty a blue folder with upload arrow, Marreta a crumbling brick wall with a red crack (`img/apps/app-*.svg`). frank-dragoon's card accent is phase-crystal teal (`#5fe0d0`), matching @@ -197,7 +197,7 @@ state — the only animation left is the speaking-voice pulse. The result is flat navy panels on crisp 1px borders: same Protoss direction, half the pixels. -**Model ids official (2026-10-08):** the card ids `frank-zealot`, +**Model ids official (2026-10-08):** the card ids `frank-zealot` `frank-dragoon`, `frank-archon` are now the harness's official model ids in `models.yml`/`config.yml` (see [[OMP Harness]]); the systemd services keep the `frank-archon`/`frank-dragoon`/ @@ -205,7 +205,7 @@ systemd services keep the `frank-archon`/`frank-dragoon`/ one name per model. **Api ids renamed on the wire (2026-10-09):** the promise in -`docs.html` is kept — the served model ids are now `frank-zealot`, +`docs.html` is kept — the served model ids are now `frank-zealot` `frank-dragoon`, `frank-archon`, so the docs page no longer carries `api id:` annotations, and the curl/opencode/Pi examples send `frank-zealot`. The systemd services keep the old names. `docs.html` @@ -216,7 +216,7 @@ ids, which did not change. clip-paths replaced with `border-radius` (the clipped corners read as gaps in the 1px border); section-title diamonds moved from absolute positioning to a fixed-width inline glyph (aligned, wrap-safe); closed -WIP items grey out (dim type, desaturated art, muted border via +WIP items grey out (dim type, desaturated art, muted border `:has(.badge.wip-closed)`); the Dragoon tagline is one spliced track (`sfx/dragoon-tagline.ogg` — loudnorm-matched, silence-trimmed, 350 ms beat; the two SC2 source oggs are trashed); privacy notes became diff --git a/serverAppConfigs-docs/Infrastructure/LAN-DNS.md b/serverAppConfigs-docs/Infrastructure/LAN-DNS.md index bcb668d..1954613 100644 --- a/serverAppConfigs-docs/Infrastructure/LAN-DNS.md +++ b/serverAppConfigs-docs/Infrastructure/LAN-DNS.md @@ -188,7 +188,7 @@ sudo systemctl restart systemd-resolved && resolvectl flush-caches ``` The `~` (route-only) prefix is load-bearing: it routes only those domains to -Pi-hole. Verify: `getent hosts www.franksdank.cloud` → `192.168.0.143`; +Pi-hole. Verify: `getent hosts www.franksdank.cloud` → `192.168.0.143` `resolvectl status` shows `DNS Domain: ~franksdank.cloud ~franksdank.casa`. **Layer 2 — browser DoH (the bypass: "curl works, browser times out").** If @@ -236,7 +236,7 @@ Endpoints (all POST, JSON body): `.content.records[]` as some docs claim — parse with `jq -r '.records[] | [.id,.name,.type,.content,(.ttl|tostring)] | @tsv'`. - Delete: `/dns/delete/{domain}/{id}` (body carries apikey+secretapikey). -- Edit: `/dns/edit/{domain}/{id}` — pass the full record (`id`, `name`, +- Edit: `/dns/edit/{domain}/{id}` — pass the full record (`id`, `name` `type`, `content`, `ttl`, `prio`). Quirks that bite (verified live 2026-08-31): diff --git a/serverAppConfigs-docs/Infrastructure/Loki.md b/serverAppConfigs-docs/Infrastructure/Loki.md index fe374d9..16a2727 100644 --- a/serverAppConfigs-docs/Infrastructure/Loki.md +++ b/serverAppConfigs-docs/Infrastructure/Loki.md @@ -55,7 +55,7 @@ The [[Alloy]] agent on Dankclaw is wired to ship container and journal logs to ` The service answers and the API works; there is simply no data in it. Measured on 2026-10-18: - `curl -s http://127.0.0.1:3100/loki/api/v1/labels` → `{"status":"success"}` with no `data` key (no label names exist), and a 7-day `query_range` on `{job="systemd-journal"}` returns `"result":[]`. -- `du -sh /opt/loki/data` → a few hundred KB of directory scaffolding (`wal`, +- `du -sh /opt/loki/data` → a few hundred KB of directory scaffolding (`wal` `chunks`, `compactor`, `tsdb-shipper-active`, `tsdb-shipper-cache`) — `find /opt/loki/data -type f` counted **8 files**, all WAL segments, cluster-seed and delete-request bookkeeping, **no chunk objects** diff --git a/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md b/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md index 97e9823..3e91553 100644 --- a/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md +++ b/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md @@ -21,7 +21,7 @@ committed. | `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules | | `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) | -Targets and datasources reference **container names** (`grafana:3000`, +Targets and datasources reference **container names** (`grafana:3000` `loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never loopback-published ports; `alloy` is **not scraped at all** (no job in the config — its `127.0.0.1:12345` UI is only probed by the homepage timer, @@ -47,7 +47,7 @@ from `deploy_observability`: - `copy_grafana_provisioning` → `/opt/grafana/provisioning/` (`:286-300`) `scripts/deploy-plan.py` maps `shared/prometheus/**` to `prometheus` (plus -`alertmanager` when the changed file is `alertmanager.yml`, `:251-256`), +`alertmanager` when the changed file is `alertmanager.yml`, `:251-256`) `shared/loki/**` to `loki` and `shared/alloy/**` to `alloy` (`:257-260`), and `shared/grafana/**` to `grafana` (`:261-264`); changes under `dankclaw/observability/` select the touched unit and its dependency consumer @@ -71,9 +71,9 @@ HTTP probe, not a metrics endpoint. **Add an alert/recording rule** — edit `shared/prometheus/rules/llm-fleet.yml`. Current delivered set: **4 alert rules + 2 recording rules** — -`LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`, +`LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff` `LlmEndpointDown`, plus `job:llm_prefix_cache_reuse:rate5m` and -`job:llm_spec_accept_length:rate5m` (`shared/prometheus/rules/llm-fleet.yml:32-110`), +`job:llm_spec_accept_length:rate5m` (`shared/prometheus/rules/llm-fleet.yml:32-110`) confirmed live: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18) reports 4 alerting + 2 recording. Rules reload with the prometheus service. @@ -101,7 +101,7 @@ curl -s -u admin: http://127.0.0.1:3002/api/datasources | python3 -m json.to The two Prometheus queries were run on 2026-10-18 and are what every number on this page comes from: 4 alerting + 2 recording rules, and `up` for -`prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`, +`prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki` `frank-zealot`, `frank-dragoon`, `frank-archon` with the four structural downs listed in [[Prometheus]]. The Grafana query needs the local admin password and was not run here; the filesystem side (`ls /opt/grafana/provisioning/*`) is @@ -114,12 +114,12 @@ table (refused = outage, 404/HTML = config defect). ## Current delivered state (delivery chain live 2026-09-17, re-verified 2026-10-18) -Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`, +Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container` loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach via `host.containers.internal`) → signal-cli, addressed by number `+12062476336` (`scripts/signal-relay.py:36-43`). Every hop re-checked -2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093`, +2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093` `curl -s http://192.168.0.143:18093/health` → `ok`, and `journalctl --user -u signal-relay` ends with successful `sent to +12062476336:` lines for `LlmEndpointDown`, `LlmRequestsQueueing`, `LlmSpecDecodeOff` and @@ -148,7 +148,7 @@ state lives in Prometheus' process, not on disk — a restart re-stamps `activeAt` on a long-standing alert, so a "fresh" alert after a reboot is not evidence of a fresh fault. (Alertmanager by contrast **does** persist: `--storage.path=/alertmanager` on a host volume keeps silences and the -notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42`, +notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42` `/opt/alertmanager/data`.) Remaining known downs are the structural ones above (HTML-scraped probes, Gitea diff --git a/serverAppConfigs-docs/Infrastructure/Observability.md b/serverAppConfigs-docs/Infrastructure/Observability.md index 41de35e..b8f3bea 100644 --- a/serverAppConfigs-docs/Infrastructure/Observability.md +++ b/serverAppConfigs-docs/Infrastructure/Observability.md @@ -51,13 +51,13 @@ Dankserver runs one member of this family, node-exporter, as its own unit ## Exposure The five HTTP surfaces publish on `127.0.0.1` only — `ss -ltnp` (2026-10-18) -shows `127.0.0.1:9090`, `127.0.0.1:9093`, `127.0.0.1:3002`, `127.0.0.1:3100`, +shows `127.0.0.1:9090`, `127.0.0.1:9093`, `127.0.0.1:3002`, `127.0.0.1:3100` `127.0.0.1:12345` and no wildcard — because each of them is unauthenticated inside, and several carry lifecycle or query APIs. Browser access goes through Caddy: each vhost (`dankclaw/Caddyfile:128-145`) proxies to the guard `anubis:8923`, and the post-auth handlers route by container name on -`dankclaw_default` — `loki.franksdank.cloud` → `loki:3100` (`:364-366`), -`grafana.franksdank.cloud` → `grafana:3000` (`:374-376`), +`dankclaw_default` — `loki.franksdank.cloud` → `loki:3100` (`:364-366`) +`grafana.franksdank.cloud` → `grafana:3000` (`:374-376`) `prometheus.franksdank.cloud` → `prometheus:9090` (`:379-381`). Alertmanager has no vhost at all. node-exporter is the exception that must live in the host network namespace, and it binds exactly two addresses rather than @@ -70,13 +70,13 @@ and the podman bridge subnet `10.89.0.0/16` before the DROP ## Deployment -`./scripts/deploy.sh dankclaw --service ` runs `deploy_observability`, +`./scripts/deploy.sh dankclaw --service ` runs `deploy_observability` which installs the unit, copies that service's config -(`copy_prometheus_config`, `copy_alertmanager_config`, `copy_loki_config`, +(`copy_prometheus_config`, `copy_alertmanager_config`, `copy_loki_config` `copy_alloy_config`, `copy_grafana_provisioning`, plus `install_signal_relay` with the alertmanager deploy), then waits on the loopback -health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy`, -`/grafana`: `/api/health`, `/loki`: `/ready`, `/alloy`: `/-/healthy`, +health endpoint (`/prometheus`: `/-/healthy`, `/alertmanager`: `/-/healthy` +`/grafana`: `/api/health`, `/loki`: `/ready`, `/alloy`: `/-/healthy` `/node-exporter`: `/metrics`). Startup order follows the data dependencies — `grafana` after `prometheus`, `alloy` after `loki` (`grafana.container:32-33`, `alloy.container:38-39`). See diff --git a/serverAppConfigs-docs/LLM/DFlash DSpark Speculative Decoding.md b/serverAppConfigs-docs/LLM/DFlash DSpark Speculative Decoding.md index 134b851..0e193ef 100644 --- a/serverAppConfigs-docs/LLM/DFlash DSpark Speculative Decoding.md +++ b/serverAppConfigs-docs/LLM/DFlash DSpark Speculative Decoding.md @@ -100,7 +100,7 @@ The DFlash drafter proposes, then ngram-mod runs as an additional implementation `--spec-type` takes a comma list, and the gate is `ctx_dft` in `common/speculative.cpp` (`common_speculative_init_result`): **draftless** types (the ngram family — simple, map-k, map-k4v, mod, cache) are added -unconditionally, while **context-backed** types (`draft-mtp`, `draft-dflash`, +unconditionally, while **context-backed** types (`draft-mtp`, `draft-dflash` `draft-dspark`, `draft-eagle3`) all require a draft CONTEXT. Consequences: - A chain may hold ONE context-backed drafter plus any number of ngram types. @@ -201,7 +201,7 @@ Speculative decoding is lossless under rejection sampling: the target verifies e On the mid-August 2026 Dankclaw shape `draft-dflash` was the benchmarked winner at 4k — 20.2 t/s, against 16.9 with ngram layered on and 8.7 ngram-only; those three figures are that retired llama.cpp era's property ([[frank-dragoon]] owns -the era and its shape). MTP is a viable alternative to test, not a bug fix: the parallel-stream releases investigated on 2026-08-13 were client-side aborts and a Dankserver crash, not speculative decoding behavior (see [[deployment]]). +the era and its shape). MTP is a viable alternative to test, not a bug fix: the parallel-stream releases investigated on 2026-08-13 were client-side aborts and a Dankserver crash, not speculative decoding behavior (see ). ## Draft Models @@ -257,7 +257,7 @@ DSpark took `ROCm error: an illegal memory access` inside the shared draft context with two concurrent 16k prompts; DFlash corrupted multi-slot into premature EOS and was retired first. The timeline, the crash class, the "concurrency is the trigger, not size" finding and the `PARALLEL=1` workaround -are on [[qwen3.8-spec-matrix]]; the shape they were measured on is +are on ; the shape they were measured on is [[frank-dragoon]]'s mid-August 2026 era. ## A/B protocol on a live llama.cpp endpoint @@ -303,7 +303,7 @@ Protocol: Read the right counters — per request `timings.draft_n` / `draft_n_accepted` / `predicted_per_second`; cumulative `/metrics` -`llamacpp:spec_decode_num_drafts_total` / `..._accepted_tokens_total`; +`llamacpp:spec_decode_num_drafts_total` / `..._accepted_tokens_total` per-task `draft acceptance = ... (N accepted / M generated)`. `draft_n == 0` while tokens flow = the proposer is NOT engaged, whatever the flags say. A zero-draft config is a useful negative control: it proves the other config's @@ -367,7 +367,7 @@ states, so a dense-27B DFlash2 is useless for a 512-expert MoE trunk. **Step 4 — acceptance reality checks.** Treat 40–60 % as the honest band for a matched pair; the arm that anchors the bottom of it on this fleet is the -depth-7 `draft-dflash` row of [[frank-medium-llamacpp-dflash2]]'s spec table +depth-7 `draft-dflash` row of 's spec table (accepted share, rate and VRAM for every depth are in that table). Cheapest proxy for `p` at temp 0 when both models are greedy and share a tokenizer: greedy continuation agreement on identical prompts, two API calls, no weights @@ -381,10 +381,10 @@ drafting. **Step 5 — check the attachment surface BEFORE the arithmetic.** Closed engines refuse external drafters by design: halogen-flash-server's `DRAFTER_CAP` gates `dflash2` on the loaded checkpoint and an unavailable -name is a deliberate 400 (a silent downgrade would make comparisons lie); +name is a deliberate 400 (a silent downgrade would make comparisons lie) there is no device routing in its loopback token protocol, so a drafter can never sit on another card. Pure llama.cpp DOES route the draft: -`--spec-draft-device/-devd` (default follows `--device`), `-ngld`, +`--spec-draft-device/-devd` (default follows `--device`), `-ngld` `-otd`, `-ctkd/-ctvd`, `--spec-draft-n-max` (default 3), `--spec-draft-p-min`. One process sees only the devices its backends were built with — CUDA+Vulkan is the combination that shows a CUDA card AND an AMD iGPU (HIP+CUDA cannot @@ -411,20 +411,16 @@ measurement that would settle it. - Speculative decoding helps low-QPS, memory-bound, latency-focused workloads. It helps less at high QPS. - DFlash needs a non-causal attention backend; the ROCm flash_attn backend must be validated before production use. -- The old `--swap-space` flag is retired; the live CPU-KV-offload knob pair is owned by [[RAM CPU Offloading in VLLM]]. +-The old `--swap-space` flag is retired; the live CPU-KV-offload knob pair is owned by . - `--speculative-config` is the preferred CLI. Legacy `--spec-method`, `--spec-model`, and `--spec-tokens` flags still exist. - Use `rejection_sample_method` default `strict` for lossless output. ## See Also - [[Quantization by Hardware Architecture]] -- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] -- [[Running Qwen3.6 27B FP8 on Dankclaw]] - [[Dankclaw System Specs]] - [[Dankserver System Specs]] -- [[VLLM Overview]] -- [[deployment]] - [[GGUF Compatibility]] — header parsing and custom tensor-type lock-in - [[Llama Cpp Serving Sizing]] — slot-pool and drafter KV sizing arithmetic -- [[Llama Cpp Multi-Device Splits]] — where the drafter's KV actually lands +- where the drafter's KV actually lands - [[frank-dragoon]] and [[frank-zealot]] — the single-GPU env drop-in A/B vehicle \ No newline at end of file diff --git a/serverAppConfigs-docs/LLM/Dankclaw Improvements.md b/serverAppConfigs-docs/LLM/Dankclaw Improvements.md index a671ec3..3fad616 100644 --- a/serverAppConfigs-docs/LLM/Dankclaw Improvements.md +++ b/serverAppConfigs-docs/LLM/Dankclaw Improvements.md @@ -30,8 +30,8 @@ taken in. - **Backend**: that era's production was **Vulkan0**, not ROCm. On this box ROCm wins prefill and Vulkan wins decode, and the decode gap widens with context: - the 64k A/B is [[qwen3.8-strix-optimizations]] §4, the 262k A/B is - [[qwen3.8-spec-matrix]] §High-Context Backend Comparison. The "HIP + the 64k A/B is §4, the 262k A/B is + §High-Context Backend Comparison. The "HIP out-performs Vulkan for decode" claim did not hold on the 8060S. - **Power profile**: the `/sys/class/drm/card0/device/power_profile` file does not exist on this machine (checked 2026-08-15). GPU DPM is already @@ -39,7 +39,7 @@ taken in. - **KV cache**: production ran `q8_0` K **and** `q8_0` V (both quantized). What `frank-zealot` runs now is its own Current shape — this row is not it. - **Spec**: DFlash/DSpark are retired. That era's production was `--spec-default` - (ngram-mod) with draft-mtp first — timeline on [[qwen3.8-spec-matrix]]; the + (ngram-mod) with draft-mtp first — timeline on ; the spec setting each endpoint runs today is on its own page. - **TTM**: the `pages_limit` raise that removed the ~15 GiB ceiling is recorded once on [[Dankclaw System Specs]] (Memory). @@ -54,8 +54,8 @@ taken in. That 2026-08 reading is the firmware-VGM era: the BIOS carved ~96 GiB as "dedicated" iGPU VRAM, so the kernel saw only ~30 GiB (the same split is -recorded in [[ling-3.0-flash-trial]] §Why Vulkan + GTT; the 76.2 GiB row is an -era [[frank-dragoon]] load reading). Every live number on this box — `MemTotal`, +recorded in §Why Vulkan + GTT; the 76.2 GiB row is an +era [[frank-dragoon]] load reading). Every live number on this box — `MemTotal` the ~114 GiB GTT aperture that is the real GPU budget, the 2.0 GiB dedicated carve and the `ttm` page cap — belongs to [[Dankclaw System Specs]], re-read 2026-10-18. @@ -125,9 +125,9 @@ All require root and a reboot for the modprobe change. ## Sources - [[frank-dragoon]] — measured backend/spec numbers -- [[deployment]] — deploy and update process +- deploy and update process - AMD ROCm RDNA3.5 system optimization guide (gfx1151 memory/BIOS) -- llama.cpp source: `ggml/CMakeLists.txt`, `ggml/src/ggml-hip/CMakeLists.txt`, +- llama.cpp source: `ggml/CMakeLists.txt`, `ggml/src/ggml-hip/CMakeLists.txt` `common/arg.cpp`, `common/common.h` - NotebookCheck Ryzen AI MAX+ 395 analysis @@ -135,5 +135,3 @@ All require root and a reboot for the modprobe change. - [[frank-dragoon]] - [[DFlash DSpark Speculative Decoding]] -- [[deployment]] -- [[index]] diff --git a/serverAppConfigs-docs/LLM/Dankclaw System Specs.md b/serverAppConfigs-docs/LLM/Dankclaw System Specs.md index 6b42b4b..0bd631a 100644 --- a/serverAppConfigs-docs/LLM/Dankclaw System Specs.md +++ b/serverAppConfigs-docs/LLM/Dankclaw System Specs.md @@ -32,7 +32,7 @@ Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp | Device ID | 0x1586 | | Architecture | gfx1151 (Strix Halo) | | GPU-Visible Memory | ~114 GiB GTT aperture (the budget; `mem_info_gtt_total` = 122,402,963,456 B read 2026-10-18; rocm-smi's smaller SVM carve-out is not the budget) | -| Physical RAM | 121 GiB visible to the OS (`free -g` total 121, read 2026-10-18) on a 128 GB part — 8 × 16 GB LPDDR5 ([[ling-3.0-flash-trial]] §hardware). The 64 GiB this page recorded as "corrected 2026-08-17" (from 32 GB) is stale | +| Physical RAM | 121 GiB visible to the OS (`free -g` total 121, read 2026-10-18) on a 128 GB part — 8 × 16 GB LPDDR5 ( §hardware). The 64 GiB this page recorded as "corrected 2026-08-17" (from 32 GB) is stale | ### NPU @@ -44,9 +44,9 @@ Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp | Driver | amdxdna 0.7.0, in-tree — bound since 2026-10-18 (`lspci -k -s c6:00.1` → `Kernel driver in use: amdxdna`; journal 2026-10-18 01:25:14 `Initialized amdxdna_accel_driver 0.7.0 for 0000:c6:00.1 on minor 0`) | | Firmware | 1.1.2.65 (`/usr/lib/firmware/amdnpu/17f0_11/npu_7.sbin` → `npu.sbin.1.1.2.65.zst`, loaded at boot 2026-10-18 01:25:13) | | Runtime | None installed. FastFlowLM 0.9.45 + XRT (installed 2026-08-16) was **retired 2026-09-11** with the `amd_iommu=off` decision — see [[dankclaw-iommu-translation-off]]; the package remains at `/home/dank/GitPackages/fastflowlm_0.9.45_ubuntu26.04_amd64.deb` | -| Status | Not in service — no NPU endpoint is deployed. The device itself is usable since the 2026-10-18 `iommu=pt` boot (`/dev/accel/accel0` present, `crw-rw---- root:render 261,0`); Gitea issue #3 tracks enablement. See [[FastFlowLM NPU Inference]] | +| Status | Not in service — no NPU endpoint is deployed. The device itself is usable since the 2026-10-18 `iommu=pt` boot (`/dev/accel/accel0` present, `crw-rw---- root:render 261,0`); Gitea issue #3 tracks enablement. See | -The NPU endpoint is retired (2026-09-11; [[frank-npu]]). The `frank-zealot` name now serves **Ternary-Bonsai-2-27B-PQ2_0** on the RTX 3090 via the llama.cpp PrismML fork (see [[frank-zealot]]); the Gemma4-12B QAT rung it once pointed at is retired history ([[frank-zealot]], 2026-09-17 → 10-01). +The NPU endpoint is retired (2026-09-11; ). The `frank-zealot` name now serves **Ternary-Bonsai-2-27B-PQ2_0** on the RTX 3090 via the llama.cpp PrismML fork (see ); the Gemma4-12B QAT rung it once pointed at is retired history (, 2026-09-17 → 10-01). ### ROCm @@ -80,7 +80,7 @@ The NPU endpoint is retired (2026-09-11; [[frank-npu]]). The `frank-zealot` name | Interface | Type | Speed | IP | Notes | |-----------|------|-------|----|-------| | eno1 | onboard | 2500 Mb/s (2.5 GbE) | 192.168.0.143 (DHCP) | Carried the RPC path to Dankserver during the 2026-08-17 pipeline-parallelism trial; no RPC listener runs today (`ss -ltn` shows no :50052, 2026-10-18). Default route via 192.168.0.1. | -| enx00e04cc2e1e5 | USB-C to Ethernet adapter | 1000 Mb/s | 192.168.0.116 | UNPLUGGED 2026-08-17. While present, the route to 192.168.0.235 went through this adapter (lower route metric), so all RPC traffic was capped at 1 GbE. See [[Pipeline Parallelism/Ethernet Bandwidth Math]] section 0. | +| enx00e04cc2e1e5 | USB-C to Ethernet adapter | 1000 Mb/s | 192.168.0.116 | UNPLUGGED 2026-08-17. While present, the route to 192.168.0.235 went through this adapter (lower route metric), so all RPC traffic was capped at 1 GbE. See section 0. | | docker0 | bridge | - | 172.17.0.1 | Usually down. | The L2 gateway/switch at 192.168.0.1 (MAC f0:4b:8a:65:c5:40) is at @@ -112,5 +112,3 @@ cable between claw and server is required for 10G. - [[frank-dragoon]] - [[Shared Memory Architecture Dankclaw]] -- [[frank-redo-2026-08-17]] -- [[FastFlowLM NPU Inference]] diff --git a/serverAppConfigs-docs/LLM/Dankserver System Specs.md b/serverAppConfigs-docs/LLM/Dankserver System Specs.md index 682ffa3..c8289e0 100644 --- a/serverAppConfigs-docs/LLM/Dankserver System Specs.md +++ b/serverAppConfigs-docs/LLM/Dankserver System Specs.md @@ -30,7 +30,7 @@ read 2026-10-18). | Property | Value | |----------|-------| | Board | **ASUS ProArt B850-Creator WiFi NEO** (Rev 1.xx) — swapped in 2026-09-20 | -| Previous | ASUS TUF GAMING B650-E WIFI (card #2 was behind the chipset there; see [[Dual-R9700-Cooperation]]) | +| Previous | ASUS TUF GAMING B650-E WIFI (card #2 was behind the chipset there; see ) | | GPU lanes | CPU GPP ports bifurcated **x8/x8** at Gen5: `PCIEX16(G5)_1` → `00:01.1`, `PCIEX16(G5)_2` → `00:01.3` (both bridges: `current_link_width=8`, `current_link_speed=32.0 GT/s`, read 2026-10-18); ~31.5 GB/s per card. Each card sits behind an on-board PCIe switch (`01:00.0`/`02:00.0`, `05:00.0`/`06:00.0` — "Navi 10 XL Upstream/Downstream Port of PCI Express Switch"), so the GPU endpoint itself trains **x16** Gen5 while the CPU-facing leg stays x8 | | Atomic routing | `AtomicOpsCap: Routing+` on every bridge in both GPU chains, `AtomicOpsCtl: ReqEn+` on both endpoints — the condition that ruled out TP=2 for the previous board (2026-09-20 census; not re-readable today — `lspci -vv` needs root) *(unverified 2026-10-18)* | | Free | `PCIEX16(G4)` (chipset, x4), M.2_2, M.2_3 | @@ -55,7 +55,7 @@ read 2026-10-18). | Architecture | gfx1201 (RDNA4) | | VRAM | 32,624 MiB each (`rocm-smi` reports 34,208,743,424 B per card) | | Link | PCIe 5.0, endpoint-trained **x16** each behind the on-board switch; the CPU-facing leg is x8 (`00:01.1` / `00:01.3`, both `current_link_width=8` at 32.0 GT/s, read 2026-10-18). No XGMI. Separate IOMMU groups — `03:00.0` → group **15**, `07:00.0` → group **20** (`/sys/bus/pci/devices/.../iommu_group`, read 2026-10-18) | -| P2P | **Live and in use.** The running engine prints "I device: the 2 cards reach each other's memory directly (peer to peer)" (`podman logs frank-archon`, 2026-10-15 boot). The 2026-09-20 census had predicted the opposite — ACS `ReqRedir+` on both root ports, so traffic hairpinning at the root complex — and that prediction is superseded; see [[Dual-R9700-Cooperation]] | +| P2P | **Live and in use.** The running engine prints "I device: the 2 cards reach each other's memory directly (peer to peer)" (`podman logs frank-archon`, 2026-10-15 boot). The 2026-09-20 census had predicted the opposite — ACS `ReqRedir+` on both root ports, so traffic hairpinning at the root complex — and that prediction is superseded; see | | PCI | 03:00.0 and **07:00.0** (were `03:00.0` / `0b:00.0` on the B650-E) | | iGPU | Raphael (gfx1036) at **11:00.0** — enumerates as ROCm device 2 (rocm-smi GPU[2], 512 MiB). The retired llama.cpp trios hid it with `HIP_VISIBLE_DEVICES=0,1`; the live radiance engine instead enumerates all three AQL devices and ignores it ("3 devices [0:gfx1201 1:gfx1201 2:gfx1036], using 2 of 3, ignoring gfx1036") | @@ -104,8 +104,7 @@ RCCL-backed rebuild needs no new host packages. | Serving stack | radiance 1.0.13 container (`stilldeadcode/radiance@sha256:4c3f484564…`), TP=2 across both R9700s — [[frank-archon]]. The llama.cpp `fb27a525` build at `~/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714` is the retired frank-medium artifact | The stale AMD-repo ROCm 6.3.3 packages (167, `~24.04`-suffixed) that used to -hijack `libhsa-runtime64` were purged on 2026-09-15 — see -[[frank-medium-rocmfpx]]. +hijack `libhsa-runtime64` were purged on 2026-09-15 — ### Active Services @@ -117,12 +116,12 @@ host-network podman container answering `curl 127.0.0.1:7770/health` → The services this page once called active are gone: `frank-medium.service` (llama-server, 32:33 layer split + DFlash2 drafter) is `not-found` — that rung -ran 2026-09-16 → 09-20 and is documented in [[frank-medium-llamacpp-dflash2]]; +ran 2026-09-16 → 09-20 and is documented in `frank-zealot.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU swap. Its archived unit was moved to `~/.AGENTTRASH/20260915-frank-zealot-retire/host-units/` and the vllm-radiance replica trio to `~/.AGENTTRASH/20260916-vllm-radiance-retire/` — neither path exists today (checked 2026-10-18), so the archive location is -unverified. See [[deployment]]. +unverified. See . ## Hardware Relevance @@ -134,7 +133,7 @@ unverified. See [[deployment]]. tokens at 32:33, ~6.3/~5.2 GiB headroom) is history. Either way a batch or context change needs a re-fit and not a guess. - 246 GiB system RAM is **not** headroom: the live shape pins a 43.88 GiB host - pool per rank (`FRANK_ARCHON_HOST_POOL_MIB=44928`, + pool per rank (`FRANK_ARCHON_HOST_POOL_MIB=44928` `scripts/frank-archon.env:184`) plus the 47.68 GiB n-gram/PLE table in RAM, and `free -g` showed **90 GiB available** with the engine loaded (read 2026-10-18). The frank-medium claim that "nothing in the serving path is @@ -147,6 +146,6 @@ unverified. See [[deployment]]. ## See Also - [[frank-archon]] — the active serving runbook -- [[Dual-R9700-Cooperation]] — why the PCIe path matters, and what the swap changed -- [[deployment]] — the deploy flow +- why the PCIe path matters, and what the swap changed +- the deploy flow - [[Shared Memory Architecture Dankclaw]] — for contrast, Dankclaw's unified memory diff --git a/serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md b/serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md deleted file mode 100644 index 4eef7ca..0000000 --- a/serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md +++ /dev/null @@ -1,327 +0,0 @@ ---- -type: reference -title: Dual R9700 Cooperation -description: "Why two R9700s could not run tensor-parallel on the old board, and what the board swap unblocked." -tags: - - frank-medium - - rocm - - rccl - - pcie-atomics - - tensor-parallelism - - dankserver -timestamp: 2026-09-16 ---- - -# Dual R9700 Cooperation - -Dankserver's two R9700s run **one tensor-parallel engine across both cards** — -today that is `frank-archon` under radiance 1.0.13, TP=2 -([[frank-archon]]). When this page was written the endpoint -was `frank-medium`: first llama.cpp's layer split -([[frank-medium-llamacpp-dflash2]], 2026-09-16 → 09-20), then from 2026-09-20 -one vLLM-radiance process at `tensor_parallel_size 2` -([[frank-medium-radiance-mxfp4]], 2026-09-20 → 09-26). This page records why -true tensor parallelism was impossible on the previous board (TUF GAMING B650-E -WIFI), and what the 2026-09-20 board swap changed. - -## The short answer - -**The wall is gone.** On the B650-E, card #2 sat behind the chipset, whose -bridges could not route PCIe atomic operations. RCCL's kernels carry a hidden -hostcall-buffer parameter and ROCr refuses to supply it to a device without -atomics, so *no* RCCL collective or P2P operation could be dispatched on that -card — see the [measured matrix](#measured-matrix) and the -[2026-09-16 table](#the-same-gate-as-this-board-reports-it-2026-09-16) below. - -On 2026-09-20 the board became an **ASUS ProArt B850-Creator WiFi NEO**, which -bifurcates the CPU's GPP ports **x8/x8**: the board-swap path this page -recommended, at B850 rather than X870E money. Both cards now hang off CPU root -ports (`00:01.1`, `00:01.3`), every bridge on both chains reads -`AtomicOpsCap: Routing+`, and **both endpoints read `AtomicOpsCtl: ReqEn+`** — -the exact precondition this page's own test plan named, and it is now -[measured](#measured-verdict-2026-09-20) rather than assumed. - -What that bought: - -- **Both cooperation modes now work, and they were both run.** vLLM TP=2 booted - and served (its R4D allreduce needs no RCCL collective at all), and llama.cpp - `--split-mode tensor` booted and served against an RCCL-backed build — so a - collective HAS been dispatched on this host since the swap. -- **TP=2 is the production shape.** It wins on every axis measured, by 2.1x on - decode and 3.6x at 4-way concurrency, and it replaced the layer split on the day - of measurement. The payload has moved since: the MXFP4 `frank-medium` shape is - retired 2026-09-26, and what runs today is the iq4r publisher under radiance - 1.0.13 ([[frank-archon]]). -- **The drafter, not the collective, is what llama.cpp lost.** Its tensor mode - could not carry the DFlash2 drafter, which is the honest caveat on the - comparison below. -- **The bandwidth story moved too.** Both cards ride Gen5 x8 (~31.5 GB/s each): - card 0 effectively unchanged (was Gen4 x16), card 1 up ~16x (was Gen4 x1, - ~1.97 GB/s). The M.2→PCIe riser plan is dead — `PCIEX16_2` holds card #2 now - and M.2_2 is free. - -## The gate, in source - -```cpp -void Device::checkAtomicSupport() { // rocclr/device/rocm/rocdevice.cpp:181 - std::vector link_attrs; - link_attrs.push_back(std::make_pair(LinkAttribute::kLinkAtomicSupport, 0)); - if (findLinkInfo(cpu_agent_info_->fine_grain_pool, &link_attrs)) { - if (link_attrs[0].second == 1) { - info_.pcie_atomics_ = true; // no else branch - } - } -} -``` - -The flag is a **queried hardware property**, consumed at -`rocvirtual.cpp:3954` (the hostcall/AQL dispatch path), in `rocblit.cpp` (SDMA -copy paths) and exported as `hipDeviceProp.hostNativeAtomicSupported`. There is -no environment variable, module parameter, or sysfs toggle around it — verified -by grepping ROCclr for `getenv`/`HSA_*`/`ROCR_*`/`AMD_*` in the gate's -neighborhood: none. Writing the endpoint's Device Control 2 bit 6 by hand -(`setpci -s 0b:00.0 CAP_EXP+28.w=0440`) does nothing because the value is -consumed at driver probe time. - -What has to be true upstream of it: every PCIe bridge on the GPU's path to the -CPU must advertise `AtomicOpsCap: Routing+`. On this board card #2's path -includes `05:00.0` and `06:09.0`, both `Routing-` — a **read-only capability -bit**, so no software can enable it. - -### The same gate, as this board reports it (2026-09-16) - -`lspci -vv` on dankserver. Card 0's chain is root port `00:01.1` → `01:00.0` → -`02:00.0` → `03:00.0`; card 1's is `00:02.1` → `05:00.0` → `06:00.0` → -`06:09.0` → `09:00.0` → `0a:00.0` → `0b:00.0`: - -| Device | Role | `AtomicOpsCap` | `AtomicOpsCtl` | Link | -|---|---|---|---|---| -| `00:01.1` | CPU root port → PCIEX16_1 | Routing+ | — | x16 | -| `03:00.0` | **card 0** | 32bit+ 64bit+ | **ReqEn+** | x16 | -| `00:02.1` | CPU root port → chipset | **Routing+** | — | x4 | -| `05:00.0`, `06:00.0`, `06:09.0` | B650 bridges | **Routing−** | — | x1 at `06:09.0` | -| `09:00.0`, `0a:00.0` | PCIe switch ports | Routing+ | — | x1 (downgraded) | -| `0b:00.0` | **card 1** | 32bit+ 64bit+ | **ReqEn−** | x16 | - -Three things fall out of that table, none of them in the source alone: - -- **The CPU side is fine** — `00:02.1` is `Routing+`. The wall is inside the - chipset, not the platform, which is why a slot on CPU lanes is the fix. -- **The endpoint's own bit differs between the cards, and that is the kernel's - doing.** `pci_enable_atomic_ops_to_root()` walks the bridge chain and refuses - to set Requester Enable when any bridge cannot route atomics. ROCclr's - `pcie_atomics_` and the endpoint's `ReqEn` are two faces of one cause, and it - makes `lspci -vv -s | grep AtomicOpsCtl` a one-command diagnosis: on - this host card 0 reads `ReqEn+` and card 1 `ReqEn−`. -- **Card 1's effective link is x1**, not the x16 its own capability block - advertises: `06:09.0` trained at x1, so ~2 GB/s is the real ceiling. That is a - bandwidth fact rather than an atomics one, and it only matters for - collectives — a 27B TP=2 allreduce is ≈5.6 ms/token at x1 against ≈1.4 ms at - x4. The layer split's boundary copy does not care. - -## Re-measured 2026-09-20 (ProArt B850-Creator WiFi NEO) - -`lspci -vv` after the board swap. Card 0's chain is `00:01.1` → `01:00.0` → -`02:00.0` → `03:00.0`; card 1's is `00:01.3` → `05:00.0` → `06:00.0` → -`07:00.0`. **No chipset bridge is in either path** — `00:02.1` now carries only -LAN/SATA/WiFi: - -| Device | Role | `AtomicOpsCap` | `AtomicOpsCtl` | Link | -|---|---|---|---|---| -| `00:01.1` | CPU GPP → PCIEX16_1 | Routing+ | — | Gen5 x8 | -| `01:00.0` | card 0 switch upstream | Routing+ | — | Gen5 x8 | -| `02:00.0` | card 0 switch downstream | Routing+ | — | Gen5 x16 | -| `03:00.0` | **card 0** | 32bit+ 64bit+ | **ReqEn+** | Gen5 x16 (on-card) | -| `00:01.3` | CPU GPP → PCIEX16_2 | Routing+ | — | Gen5 x8 | -| `05:00.0` | card 1 switch upstream | Routing+ | — | Gen5 x8 | -| `06:00.0` | card 1 switch downstream | Routing+ | — | Gen5 x16 | -| `07:00.0` | **card 1** | 32bit+ 64bit+ | **ReqEn+** | Gen5 x16 (on-card) | - -- Both endpoints read `CAP_EXP+28.w = 0x1440` — bit 6 set, which is the - kernel's `pci_enable_atomic_ops_to_root()` having walked a chain with no - `Routing-` bridge. Same one-command diagnosis, opposite answer. -- `dmesg` agrees on bandwidth: `252.056 Gb/s available PCIe bandwidth, limited - by 32.0 GT/s PCIe x8 link at 0000:00:01.1` (and `.13`) — ~31.5 GB/s per - card. DMI confirms the bifurcation: `PCIEX16(G5)_2`, "8x or x8". -- `rocm-smi --showtopo` reported `PCIE` between the cards (Navi 48 has no XGMI), - they were in separate IOMMU groups (15 and 20 — re-read 2026-10-18: still 15 - and 20), and both root ports had ACS `ReqRedir+`, so this page inferred P2P - would hairpin at the root complex. **That inference is superseded:** the live - engine reports the two cards reaching each other's memory directly, which - [[frank-archon]] and [[Dankserver System Specs]] record. Plain P2P traffic - measured 45.27 GB/s bidirectional on the *old* board — DMA was never the gate, - atomics were. -- Kernel 7.0.0-31-generic with `iommu=pt` then; this host runs 7.0.0-38-generic - with `iommu=pt` today (`/proc/cmdline`, read 2026-10-18). Both were ruled out - as causes ([above](#ruled-out-empirically-do-not-re-derive)) and this host's - IOMMU posture is unchanged. - -**Still unproven (as of 2026-09-20).** The gate that matters is ROCclr's -`checkAtomicSupport()` reading the `kLinkAtomicSupport` link attribute; the ReqEn -bit is the same cause seen from the kernel side, but no RCCL kernel had been -launched since the swap. Until the 2-rank probe runs — or a `--split-mode row` -boot that does *not* log `internal AllReduce init failed ... falling back to -meta-backend butterfly` — "collectives work here" was a strong expectation, not a -measurement. **RCCL never got that measurement**; what settled the operational -question instead is production traffic — radiance's TP=2 wire carries cross-rank -all-reduces over the direct P2P path and prints its own precision, and the -measured `wht6` banner with its exact-step bound is on [[frank-archon]]. - -## Measured matrix - -Bare 2-rank `torch.distributed` (`backend=nccl`) on the two R9700s, no vLLM, no -model. Rank 0 = card #1 (CPU lanes), rank 1 = card #2 (chipset): - -| Operation | rank 0 (card #1) | rank 1 (card #2) | -|---|---|---| -| `broadcast` | OK | **FAIL** `DistBackendError` | -| `all_reduce` | OK | **FAIL** | -| `barrier` | OK | **FAIL** | -| `send` / `recv` (P2P) | blocked, then timeout | **FAIL** | -| `all_gather` (after failure) | group unusable | — | - -The `AMD_LOG_LEVEL=3` trace names the cause: `Pcie atomics not enabled, hostcall -not supported` → `AQL dispatch failed!` → `hipLaunchKernel` returns -`hipErrorIllegalState` for `ncclDevKernel_Generic_4(...)`. Nothing in vLLM or -RCCL mentions atomics; the runtime log is the only place it appears. - -**`send`/`recv` failing is the decisive result**: it kills pipeline parallelism -as an option, because vLLM's PP stage handoff is exactly that primitive. - -## Software escapes: closed, and the last one is now moot - -Every row below was written under the old board. The last row — the only door -that was ever open — stopped mattering on 2026-09-20, because the hardware gate -it existed to route around no longer applies to this host. - -| Idea | Finding | -|---|---| -| vLLM's IPC custom allreduce | **Gated off by a table keyed to CUDA sm versions** — `{9.0, 10.0, 10.3, 10.7}`; ROCm reports capability `12.0` for gfx1201, so the gate is `False`. `custom_ar` itself is compiled into the image. | -| ...and if that table were patched | Still insufficient: `broadcast`, `barrier`, and the TP logits `all_gather` have no non-RCCL path. | -| The radiance fork's own allreduce | Already present and *on by default* (`RADIANCE_USE_R4D_AR`, size gate `RADIANCE_AR_MAX_KB`): a one-shot P2P push allreduce wrapping `CudaCommunicator.all_reduce`, exchanging IPC handles over the **gloo** group. With the gate raised it covers prefill and decode, and `ncclDevKernel` disappears from the profile. But it does not intercept `PyNcclCommunicator.__init__`'s **warmup allreduce**, nor `pynccl_comm.all_gather` — both direct RCCL calls. | -| NCCL symmetric memory | Unavailable: the shipped RCCL advertises NCCL 2.30.4 but exposes no `ncclDevComm*` API, so torch's symmetric-memory path cannot run. | -| Prefill/decode disaggregation | Needs a KV-transfer connector plus coordination collectives; both out. | -| gloo for GPU work | CPU/TCP collectives — orders of magnitude too slow for per-layer TP. | -| **Rebuild RCCL so its kernels do not declare a hostcall buffer** | **OPEN — unverified.** The claim is that the hostcall requirement is triggered by device-side `printf`/`assert`, and that `-mprintf-kind=buffered` avoids it. Counter-evidence: RCCL's device source at `src/device/` contains **no live** `printf`, `assert`, or `__assertfail` (checked), so what makes its kernels declare `HiddenHostcallBuffer` is unexplained. The claim traces to search-result summaries, not a primary source. Cheap to settle once a host is available: recompile librccl with `-mprintf-kind=buffered`, dump a kernel's declared arguments (`roc-obj-extract` + `llvm-readelf --notes`) and look for `hidden_hostcall_buffer`; if it is gone, a 2-rank probe should dispatch. **Stop and mask the service first** — see the note below. | - -## Bandwidth: why x4 is enough - -A 27B TP=2 allreduce at batch 4 moves roughly -$2 \times \text{batch} \times \text{hidden} \times 2\ \text{bytes} \approx 160\ \text{KB}$ -per layer per forward pass; across ~62 layers that is **≈10 MB per token-step**, -or **~1.4 ms at PCIe 4.0 x4** (~7.9 GB/s), and under 0.4 ms at the Gen5 x8 -~31.5 GB/s each card now has. TP is compute-bound here, not interconnect-bound. -That is why the cheap option (the M.2 riser) and the expensive one (the board -swap) were both considered adequate on bandwidth alone — and why the board that -landed, at x8/x8, clears the arithmetic with room to spare. - -## Ruled out empirically (do not re-derive) - -- Kernel versions 7.0.0-28 / -30 / -31 — byte-identical failure. -- `iommu=pt`, and `amd_iommu=off` is unrelated (that was a Dankclaw prefill knob). -- P2P hardware: 45.274 GB/s bidirectional between the cards works fine — plain - memory traffic is not the problem; *atomic* routing is. -- A from-source librccl rebuild: identical failure at the identical call path. -- Every NCCL transport/algorithm knob. -- A one-card test proves nothing: NCCL short-circuits at `world_size=1`, which - is exactly why the two replicas work at all. -- Independent corroboration: StillDeadcode/vllm-radiance Codeberg issue #22 — - different host, same image, kernel 7.0.0-30, same `enqueue.cc` failure. - -## Measured verdict (2026-09-20) - -Both cooperation modes were run on this host on 2026-09-20 — the same salted -10-prompt battery (`scripts/bench-conversation.py`), the service stopped and -masked between arms. The arm table is [[frank-medium-radiance-mxfp4]]'s bench of -record (arms R = radiance TP=2 + DFlash2 as deployed, I = the llama.cpp layer -split that preceded it, L = llama.cpp `--split-mode tensor` against RCCL); this -page keeps only the verdict. **TP=2 won on every axis — roughly 2.1× decode and -3.6× at 4-way concurrency over the layer split — and replaced the layer split the -day of measurement.** - -Arm L carries a caveat no flag can remove: llama.cpp's tensor mode could not load -the DFlash2 drafter (it aborts at `ggml-backend.cpp:941` — a draft weight -pre-allocated on the non-primary device has no backend once `n_copies == 1`), so -it decoded **without** speculation while the other arms decoded with it; and part -of R's margin over I is the 4-bit MXFP4 checkpoint, not the parallelism. - -Two numbers elsewhere on this page are therefore not interchangeable: - -- The 272 t/s recorded 2026-09-16 is **prefill**, on a **butterfly** allreduce, - with RCCL off, on the old board. It was never a TP decode number, and its - paired decode figure is in "The third path" below. -- The butterfly tensor arm and the RCCL tensor arm a day later are the same shape - family on different collective paths; the measured pair lives with - [[frank-medium-radiance-mxfp4]]'s arms. - -RCCL needs PCIe atomics on *every* participating GPU — the gate the old board -failed. vLLM's R4D one-shot allreduce dispatches no collective at all, which is -why TP=2 runs today rather than merely "being unblocked": the image ships TheRock -RCCL, and TheRock carries no gfx1201 device code (`invalid device function`), so -radiance could not have used RCCL on this host even with atomics routed. Ubuntu's -RCCL 2.27 does carry gfx1201 kernels, and that is what arm L linked against. - -**Rule for any future multi-GPU collective experiment: stop and mask the -production service first**, per the benchmarking policy, or run it on a host that -is not serving users. The production service is `frank-archon`, a **root** unit, -so on Dankserver that is `sudo systemctl stop frank-archon && sudo systemctl -mask frank-archon` (`scripts/frank-archon.service`; never `disable` — that -deletes the unit file, and `mask` is what the bench policy prescribes). The -2026-09-20 wording below — `systemctl --user stop frank-medium && systemctl ---user mask frank-medium`, frank-medium being a rootless quadlet — is retired -with that service, which today reports `not-found` (read 2026-10-18). - -This is not theoretical: on 2026-09-16 a bare RCCL P2P probe was run against the live -service; the host went hard down ~20 minutes later (no ARP, no ICMP, required a -power cycle). The mechanism is not proven, but a hung AQL dispatch wedging the -GPU driver is a plausible cause and this host has precedent for going -unreachable until a power cycle. - - -## The third path (production in the layer-split era): llama.cpp's layer split - -Everything above is about *collectives*, and vLLM cannot proceed without them -in any of its parallelisms — TP needs a per-layer allreduce, and PP's stage -handoff is a p2p `send`/`recv` of the hidden state, the primitive this host -also fails. **llama.cpp's `--split-mode layer` needs no collective at all**: -layers are distributed across the cards and the boundary activation is a -plain per-backend copy, once per forward pass. No RCCL kernel is ever launched -on the second card, so the atomics gate never comes up. - -That is what `frank-medium` ran from 2026-09-16 to 09-20 -([[frank-medium-llamacpp-dflash2]]): one process, one model, both cards, -262144 context. Its measured layer-split rates are that page's; this host's -tensor arm under the butterfly fallback measured 272 t/s prefill / 26.3 tok/s -decode. **That tensor number is not a prediction for an RCCL-backed build** — it -is what tensor split costs when the allreduce degrades to a meta-backend -butterfly, and it is the number the board-swap path exists to retire. - -The ranking therefore changed twice. On 2026-09-16: software already gets both -cards onto one model, so the hardware fix stopped being a capability question -and became a performance one — *tensor* parallelism, ~40 % decode, plus one -pooled KV instead of two independent 122k-token pools. On 2026-09-20 the -hardware half of that question was answered by the board swap; the llama.cpp -measurement half never ran, and `GGML_HIP_RCCL` still defaults `OFF` in -`scripts/build-llama-rdna4.sh:41` (re-read 2026-10-18) for exactly the reason -this page documents — moot while no llama.cpp shape is deployed on this host. - -## What would change the answer - -- ~~An RCCL build whose kernels stop declaring a hostcall buffer~~ — **moot - since 2026-09-20.** That door only mattered while the hardware gate could not - be passed; the board swap passes it. The `-mprintf-kind=buffered` hypothesis - and its counter-evidence stay in the escape table for the record. -- The symmetric-kernel path inside RCCL, if a release ever ships the device API - (`ncclDevComm*`) that torch's symmetric-memory path needs. The shipped TheRock - nightly (`~/.local/share/rocm-7.14.0a20260624/lib/librccl.so.1`) still does - not. -- ~~Firmware or an AGESA update that makes the chipset bridges advertise atomic - routing~~ — arrived as new silicon instead. -- A GPU-side workaround: none found in ROCclr, and the gate is per-device, - queried once at probe. - -## See Also - -- [[Pipeline Parallelism/index|Pipeline Parallelism]] — two-machine research (the PP variant of this problem) -- [[Dankserver System Specs]] — hardware inventory -- [[Observability]] — the scrape/dashboard/alert stack around these endpoints diff --git a/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md b/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md deleted file mode 100644 index e1a4edc..0000000 --- a/serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md +++ /dev/null @@ -1,448 +0,0 @@ ---- -type: reference -title: FastFlowLM NPU Inference -description: "The AMD NPU-first runtime: fit analysis for the XDNA2, published benchmarks, install state, and test runbook." -tags: - - fastflowlm - - npu - - xdna2 - - dankclaw - - research -timestamp: 2026-09-03 ---- - -> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology. -> See [[frank-redo-2026-08-17]] and [[deployment]] for the active configuration. -> The `frank-zealot` name and port 52625 then belonged to the vLLM -> service on Dankclaw (nameistoken Quark W8A8); this page is archive. -> -> **The NPU endpoint is retired — 2026-09-11.** On Dankclaw today -> `frank-npu.service` is `masked`/`inactive` and nothing listens on 52625 -> (`ss -ltn`, read 2026-10-18). The runtime is still installed -> (`fastflowlm 1.0.1`, `libxrt2`/`libxrt-npu2`, `/opt/fastflowlm`, -> `~/.config/flm`, `/usr/bin/flm` — all read 2026-10-18) and the device is -> usable again since the 2026-10-18 `iommu=pt` boot (`amdxdna` bound, -> `/dev/accel/accel0` present — [[dankclaw-iommu-translation-off]]), but no -> service runs on it. `frank-zealot` today is the llama.cpp PrismML endpoint on -> the RTX 3090 ([[frank-zealot]]). See [[frank-npu]] -> for the retirement record. Everything below is the 2026-08 research record. - -# FastFlowLM NPU Inference - -FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM was installed on Dankclaw, the NPU validated, and `frank-zealot` (port 52625) served in production — that arrangement ended with the 2026-09-11 retirement (see the banner above). See Promotion. - -## Verdict (2026-08-16) - -- The NPU is a complement, not a replacement. It fits dense models up to about 9B. -- Qwen3.8-27B (dense) cannot run on the NPU. No FLM build exists and the memory budget is too small. -- Qwen3.6-35B-A3B has an official FLM build (Q4_KS). AMD benchmarks it on a 96 GB machine. It probably does not fit the 30 GiB budget of Dankclaw. -- Realistic first candidates: Qwen3.5-9B, Qwen3-8B, gpt-oss-20b (at the budget edge). -- **Landed 2026-08-16:** gpt-oss-20b serves on the NPU at 19.3 t/s decode, concurrently with the PP head. See Measured Results. -- The memlock limit was the setup blocker. Fixed 2026-08-16 via `/etc/security/limits.d/99-flm-memlock.conf`. - -## What FastFlowLM Is - -FLM is an NPU-first inference runtime. It started as a 2025 university project (University of Rhode Island, Ken Qing Yang). The team joined AMD on 2026-07-17. The repository moved to the ROCm organization on 2026-08-11 as v1.0.0. - -Key properties, per the project README and docs: - -- Runs fully on the NPU. No GPU or CPU load during inference. -- CLI mode (`flm run `) and server mode (`flm serve `). -- Server mode exposes a REST API and an OpenAI-compatible API. Default port is 52625. -- Models are not GGUF. FLM downloads pre-compiled kernel packs (built with AMD IRON + AIE-MLIR) from Hugging Face. -- Context lengths up to 256k tokens on listed models. -- Vision, audio, embedding, and MoE model support. -- Runtime is about 17 MB. Orchestration code is MIT-licensed. Kernels are free for commercial use. -- Models store in `~/.config/flm/` (Linux). Override with `FLM_MODEL_PATH`. - -## NPU Hardware on Dankclaw - -Dankclaw runs an AMD Ryzen AI MAX+ 395 (Strix Halo). The chip integrates an XDNA2 NPU -with 50 TOPS of compute and, crucially, **no dedicated VRAM** - it shares system memory -via the IOMMU SVA mapping. The budget this page reasons about - 29 GiB usable at -4800 MT/s LPDDR5X - is the 2026-08 host state; the live figure is [[Dankclaw System Specs]]. - -Live verification on Dankclaw, 2026-08-16: - -``` -$ flm validate -[FLM] Fetching models from: /opt/fastflowlm/share/flm/model_list.json -[Linux] Kernel: 7.0.0-29-generic -[Linux] NPU: /dev/accel/accel0 with 8 columns -[Linux] NPU FW Version: 1.1.2.65 -[Linux] amdxdna version: 0.7 -[ERROR] Memlock limit is too low (8MB). Please raise the limit or set to infinity. -``` - -- Kernel 7.0.0-29-generic shipped the `amdxdna` driver in-tree, loaded. The host now runs 7.0.0-38-generic and the driver is still bound (`lspci -k -s c6:00.1` → `Kernel driver in use: amdxdna`, read 2026-10-18). -- Firmware 1.1.2.65 meets the 1.1.0.0 minimum. -- The XRT runtime packages (`libxrt2`, `libxrt-npu2`) are installed — still installed on 2026-10-18 (`dpkg -l` → `libxrt2 1:2.21.75+dfsg-4`, `libxrt-npu2 1:2.21.75+dfsg-4`). -- The only failure is the memlock ulimit (8 MB). FLM needs `unlimited`. -- Ubuntu 26.04 is not on the FLM supported-distro list (24.04, 25.10, Arch, generic). Validation passes anyway. Treat 26.04 as "generic Linux, working". - -## Memory Model - -The NPU allocates from system RAM. AMD/Microsoft enforce an internal cap of about half of total system RAM for NPU use. The gpt-oss model card states a cap of about 15.6 GB on a 32 GB machine, which is half. - -Budget math for this fleet: - -| Machine | RAM | NPU budget (approx. half) | -|---------|-----|---------------------------| -| Dankclaw (Strix Halo) | 30 GiB | ~15 GiB | -| Dankserver (Ryzen 7 7700X) | 62 GB | no NPU | -| FLM bench box (Kraken Point) | 32 or 96 GB | ~16 or ~48 GB | - -The half-of-RAM gate scales with host RAM - live figures in [[Dankclaw System Specs]] -and [[Dankserver System Specs]]. The live gate on Dankclaw is still `flm validate` -(it owns the current number). - -Weight sizes (approximate, Q4-class): - -| Model | Quant | Weights | Fits ~15 GiB budget? | -|-------|-------|---------|----------------------| -| Qwen3.5-9B | Q4_1 | ~5.5 GB | yes | -| Qwen3-8B | Q4_1 | ~5 GB | yes | -| gpt-oss-20b (MoE, 3.6B active) | Q4_1 | ~15.1 GB | at the edge | -| Qwen3.6-35B-A3B (MoE, 3B active) | Q4_KS | ~20 GB | no, unproven at 30 GiB | -| Qwen3.8-27B (dense) | any Q4 | ~16 GB+ | no build exists | - -Do not load a model larger than the budget first. An oversized load can starve the unified pool and crash the box while production services still run. - -## Supported Models (Qwen family, plus gpt-oss) - -All Qwen cards use Q4_1 unless noted. Context lengths are the FLM-published maxima. - -| Tag | Base model | Context max | Notes | -|-----|-----------|-------------|-------| -| `qwen3:0.6b` | Qwen3-0.6B | 32k | thinking toggle | -| `qwen3:1.7b` | Qwen3-1.7B | 32k | thinking toggle | -| `qwen3:4b` | Qwen3-4B | 32k | tool calling | -| `qwen3:8b` | Qwen3-8B | 32k | tool calling | -| `qwen3-tk:4b` | Qwen3-4B-Thinking-2507 | 256k | thinking | -| `qwen3-it:4b` | Qwen3-4B-Instruct-2507 | 256k | tool calling | -| `qwen3vl-it:4b` | Qwen3-VL-4B-Instruct | 256k | vision | -| `qwen2.5-it:3b` | Qwen2.5-3B-Instruct | 32k | | -| `qwen2.5vl-it:3b` | Qwen2.5-VL-3B-Instruct | 256k | vision | -| `qwen3.5:0.8b` | Qwen3.5-0.8B | 256k | vision, thinking toggle | -| `qwen3.5:2b` | Qwen3.5-2B | 256k | vision, tool calling | -| `qwen3.5:4b` | Qwen3.5-4B | 256k | vision, tool calling | -| `qwen3.5:9b` | Qwen3.5-9B | 256k | vision, tool calling | -| `qwen3.6-moe:35b-a3b` | Qwen3.6-35B-A3B | 256k | **Q4_KS**, MoE, vision, thinking, tool calling | -| `gpt-oss:20b` | gpt-oss-20b | 128k | Q4_1, ~15.1 GB in FLM | - -Other families: LLaMA, DeepSeek, Gemma, MedGemma, TranslateGemma, LiquidAI/LFM, Phi, Nanbeige, Whisper, EmbeddingGemma, SmolVLA. - -## Published Benchmarks (FLM, vendor numbers) - -All vendor results come from AMD Ryzen AI 7 350 (Kraken Point, 40 TOPS NPU) boxes under the default NPU power mode (Performance). Our NPU is a 50 TOPS Strix Halo XDNA2. Treat these numbers as directional, not as our expected speed. - -Qwen3.5, FLM v0.9.38, Kraken Point 32 GB DRAM. Decode t/s by starting context length: - -| Model | 1k | 2k | 4k | 8k | 16k | 32k | -|-------|-----:|-----:|-----:|-----:|-----:|-----:| -| Qwen3.5-0.8B | 39.2 | 38.0 | 36.3 | 33.1 | 28.1 | 21.6 | -| Qwen3.5-2B | 26.8 | 26.2 | 25.4 | 23.7 | 21.3 | 17.0 | -| Qwen3.5-4B | 15.0 | 14.6 | 14.2 | 13.3 | 11.8 | 9.6 | -| Qwen3.5-9B | 9.3 | 9.2 | 9.0 | 8.5 | 7.8 | 6.9 | - -Prefill t/s for the same run: - -| Model | 1k | 2k | 4k | 8k | 16k | 32k | -|-------|-----:|-----:|-----:|-----:|-----:|-----:| -| Qwen3.5-0.8B | 983 | 1257 | 1471 | 1584 | 1579 | 1447 | -| Qwen3.5-2B | 803 | 1004 | 1142 | 1223 | 1225 | 1151 | -| Qwen3.5-4B | 378 | 440 | 479 | 493 | 487 | 450 | -| Qwen3.5-9B | 284 | 333 | 362 | 379 | 378 | 357 | - -Qwen3.6-35B-A3B (Q4_KS), FLM v0.9.45, Kraken Point **96 GB DRAM**: - -| Metric | 1k | 2k | 4k | 8k | 16k | 32k | -|--------|-----:|-----:|-----:|-----:|-----:|-----:| -| Decode t/s | 13.65 | 13.41 | 13.09 | 12.51 | 11.24 | 9.51 | -| Prefill t/s | 78.98 | 118.04 | 156.43 | 197.93 | 218.84 | 221.96 | - -- PP and NPU together cost 2.2 GiB + NPU model + ~2 GiB of headroom. The then-current - PP split (head + 3090 worker, 2026-08 era - see [[frank-redo-2026-08-17]]) decoded at - 14-18 t/s for the 27B dense model, so an NPU offload would free the head or upgrade - the worker's model. - -The vendor also claims "over 10x more power-efficient" versus GPU. That is a marketing claim. We have not measured it. - -## Fit to the Frank Program - -| Our model | NPU verdict | -|-----------|-------------| -| Qwen3.8-27B UD-Q8_K_XL (Dankclaw, dense) | No. No FLM build. ~16 GB+ even at Q4. Exceeds the ~15 GiB budget. | -| Qwen3.6-35B-A3B (Dankserver, MoE) | Unproven. Official build exists. Vendor bench ran on 96 GB RAM. A 30 GiB box has ~15 GiB for the NPU. Test last, with the service stopped. | -| Qwen3.5-9B / Qwen3-8B | Yes, the realistic tier. ~5-6 GB. Leaves room for KV and the OS. | -| gpt-oss-20b (MoE) | **Landed.** Weights are 13.8 GB on disk (q4nx). The model holds ~15 GiB resident on the NPU. Decode is 19.3 t/s. See Measured Results. | - -Proposed role: a small, always-on, low-power endpoint for triage, summarization, embeddings, or agent side-tasks on the Dankclaw NPU. It frees the iGPU and the 3090 for the 27B/35B workloads. It does not replace either. - -## Install State on Dankclaw (verified 2026-08-16) - -- `fastflowlm 1.0.1` installed (deb, upgraded from 0.9.45-12-ga6bd056 on 2026-08-17). `flm` at `/usr/bin/flm`. The 0.9.45 runtime refused the qwythos pack, which declares `flm_version` 1.0.0. -- `libxrt2` / `libxrt-npu2` (Xilinx Runtime) installed. -- `/dev/accel/accel0` present. `flm validate` passed fully. (Re-read 2026-10-18: `/dev/accel/accel0` is still present and `fastflowlm 1.0.1` is still installed, but the endpoint is retired — `frank-npu.service` masked, no listener on 52625.) -- Memlock fixed: `/etc/security/limits.d/99-flm-memlock.conf` sets soft and hard memlock to unlimited. A fresh sshd session picks it up via pam_limits. No reboot needed. -- `gpt-oss:20b` pulled (4/4 files verified, 13.8 GB `model.q4nx` + tokenizer) into `~/.config/flm/models/GPT-OSS-20B-NPU2/`. -- `qwythos:9b` (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2) installed 2026-08-17 via `flm-add.py` (downloaded from the HF repo; stdlib-only, no pip needed). Tag registered in the user registry `~/.config/flm/model_list.json` (a copy of the v1.0.1 system registry + the custom entry); kernels symlinked `~/.config/flm/xclbins/Huihui-...-NPU2 -> /opt/fastflowlm/share/flm/xclbins/Qwen3.5-9B-NPU2`. The service env exports `FLM_CONFIG_PATH` and `FLM_XCLBIN_PATH` so the tag resolves (without them flm hits the unknown-tag fallback, see the incident below). -- `flm serve gpt-oss:20b --host 0.0.0.0 -p 52625 --pmode performance` runs as a nohup process. It is not a systemd unit. It does not survive a reboot. -- The `llama-server` (frank-dragoon, the PP head) holds an fd on `/dev/accel/accel0` from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16. - -## Measured Results (2026-08-16, gpt-oss-20b on the Dankclaw NPU) - -Conditions: `--pmode performance`, PP head (27B, port 7770) serving concurrently, 5.5 GiB system RAM headroom steady-state, NPU resident footprint ~15 GiB. - -| Run | Prompt tok | Compl tok | Wall s | Prefill t/s | Decode t/s | KV occ % | -|-----|-----------:|----------:|-------:|------------:|-----------:|---------:| -| 1 (smoke) | 74 | 63 | 7.2 | 19.5 | 19.3 | 1.7 | -| 2 (160-tok prompt) | 161 | 256 | 17.4 | 40.8 | 19.3 | 5.1 | -| 3 (95-tok prompt, 512 out) | 95 | 512 | 30.8 | 24.3 | 19.3 | n/a | - -Decode is flat at 19.3 t/s across all three runs. Output includes `reasoning_content` (gpt-oss is a reasoning model). At `max_tokens 256` one run spent its whole budget thinking and returned empty content. Give it 500+ tokens or strip reasoning in the client. - -Long-context prefill is not yet measured (runs 1-3 used under 200 prompt tokens). The 35B-A3B step remains untested. - -## Model Bakeoff (measured 2026-08-16, 4 models) - -All runs on the Dankclaw NPU (50 TOPS Strix Halo), single stream, `--pmode performance`, PP head (frank-dragoon) serving concurrently on 7770. Decode t/s is the FLM-reported rate. Vision accuracy = ground-truth elements out of 4 (text SPECTRUM, red circle left, blue square right, three green triangles bottom). - -| Model | Tag | Weights | Decode t/s (short) | 8k prefill t/s | 8k decode t/s | Vision decode t/s | Vision accuracy | Max ctx (registry) | -|-------|-----|---------|-------------------:|---------------:|--------------:|------------------:|----------------|--------------------| -| Gemma4-E4B | `gemma4-it:e4b` | 6.8 GB + 912 MB audio | 12.2 | 582 | 9.9 | 11.7 | 4/4 | 32768 | -| Qwen3.5-9B | `qwen3.5:9b` | 7.3 GB | 7.2 | 343 | 6.6 | 7.3 | 4/4 | 32768 | -| GPT-OSS-20B | `gpt-oss:20b` | 13.8 GB | 19.3 | 378 | 16.9 | n/a (no vision) | 0/4 | **8192** | -| Qwen3.6-35B-A3B | `qwen3.6-moe:35b-a3b` | 22.2 GB | 12.7 | 207 | 11.2 | 12.7 | 4/4 | 32768 | - -Notes: - -- gpt-oss:20b is the fastest decoder (MoE, 3.6B active) but its KV cache fills at ~8k tokens (98.85% occupancy at 7,926; requests past ~8k return 400). It has no vision encoder. -- 35B-A3B loads 28-29 GiB resident. It works but leaves ~1.5 GiB headroom with the PP head running. Do not run it alongside the head at full load. -- Vision prefill for the two vision models with ~720 token image input: gemma 141 t/s, qwen3.5 176 t/s, 35B-A3B 68 t/s. -- All four comply with the same image at 4/4 elements except gpt-oss, which cannot see it. - -## Promotion (2026-08-16) - -`frank-zealot` then ran the NPU endpoint on Dankclaw (2026-08; the name was reused -twice after this window - see the banner - and the NPU unit today is `frank-npu`, -masked). Unit: `scripts/frank-zealot.service` (symlinked into `/etc/systemd/system`), -wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-zealot.env`. -`LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap -`FLM_MODEL` in the env to switch. - -### Boot policy (2026-08-17) - -The `frank-zealot` unit is disabled for boot. Only the PP pair (`frank-dragoon` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw: - -``` -sudo ln -sf /home/dank/vllmScripts/scripts/frank-zealot.service /etc/systemd/system/frank-zealot.service -sudo systemctl daemon-reload -sudo systemctl enable frank-zealot -``` - -Live config (2026-08-16): `FLM_PMODE=turbo`, `FLM_ASR=1` (loads Whisper-V3-Turbo-NPU2; exposes `/v1/audio/transcriptions`), `FLM_CTX_LEN=131072` (verified max — Gemma4 natively supports 128K and the NPU budget fits it; KV occupancy measured at 0.11% for a 123-token exchange, decode 12.0 t/s, 13 GiB headroom). - -The PP split was renamed `frank-pp-head` → `frank-dragoon` (2026-08-16, commit `48142f5`). - -Model swap (2026-08-16, commit `50c55ee`): `FLM_MODEL=qwen3.6-moe:35b-a3b`, `FLM_CTX_LEN=262144` (native max), `FLM_PMODE=turbo`, `FLM_ASR=1` kept. Decode measured 12.5-13.1 t/s. The Gemma4 live config above (131072 ctx) applied to `gemma4-it:e4b`. Swap `FLM_MODEL` and `FLM_CTX_LEN` back to restore it. - -Model swap (2026-08-17, commit `c85ff6c`): `FLM_MODEL=qwythos:9b` (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2, Q4NX 7.76 GB). The pack installs through `flm-add` (user registry `~/.config/flm/model_list.json` + user xclbins `~/.config/flm/xclbins`, kernels symlinked from official `Qwen3.5-9B-NPU2`). Runtime upgraded to FLM 1.0.1 first (see Install State). `FLM_CTX_LEN=262144` native max (Qwen3.5-9B hybrid: 8 of 32 layers full attention, so KV stays small; occupancy 0.008% at 21 tokens). `FLM_PMODE=turbo`, `FLM_ASR=1` kept. The env file now exports `FLM_CONFIG_PATH` / `FLM_XCLBIN_PATH` — required, or flm treats the tag as unknown and silently serves llama3.2:1b. `qwen3.6-moe:35b-a3b` files stay installed for rollback. pi client entry (`~/.pi/agent/models.json` on dankware): id `qwythos:9b`, ctx 262144, input `[text]` (pack has no vision weights), compat `supportsDeveloperRole: false` kept. - -## Parallel Streams (measured 2026-08-16) - -The NPU serializes generations. It runs exactly one stream at a time. Extra streams queue in FIFO order. FLM server flags `--socket` (default 10) and `--q-len` (default 10) cap queue depth only. They do not add concurrency. - -Concurrent-request test, gpt-oss-20b, 128-token cap per stream: - -| Streams | Wall s | Aggregate t/s | Per-stream wall s | -|--------:|-------:|--------------:|-------------------| -| 1 | 10.6 | 12.1 | 10.6 | -| 2 | 21.1 | 12.0 | 10.8 / 21.1 | -| 4 | 41.1 | 11.6 | 10.9 / 21.6 / 32.4 / 41.1 | - -The 4-stream wall times form a near-perfect 10.7 s ladder. Each stream self-reports 19.3 t/s decode, counting only its own active slices. Trust per-stream wall time, not the self-reported speed, under concurrency. - -Consequence: the NPU endpoint is a single lane. It suits low-frequency background work (triage, summarization, agent side-tasks). It does not suit interactive chat for several simultaneous users. The 27B llama.cpp service, by contrast, runs PARALLEL=2 with true concurrent slots. - -## Test Runbook -This runbook targets a retired endpoint; keep it as the procedure if the NPU is -ever revived. - -**Host RAM availability.** The gate is half of system RAM by default - a relic of the -install: the endpoint ran on a machine that presented 30 GiB of system RAM; on today's -box that ceiling is far larger (live figure: [[Dankclaw System Specs]]). Check free RAM -first, or stop -the service the way the bench policy requires (`sudo systemctl stop -frank-dragoon && sudo systemctl mask frank-dragoon` — self-healing would -otherwise restart it). The NPU shares the unified pool with everything else on -the box. - -```mermaid -graph TD - A[1. Raise memlock to unlimited] --> B[2. flm validate + xrt-smi examine] - B --> C{NPU visible to XRT?} - C -- no --> D[Fix driver or firmware path] - C -- yes --> E[3. flm run qwen3.5:4b smoke test, /verbose] - E --> F{Generates tokens?} - F -- no --> D - F -- yes --> G[4. flm serve on a scratch port, never 7770] - G --> H[5. Bench decode + prefill t/s] - H --> I{Speed acceptable?} - I -- no --> J[Record results, stop] - I -- yes --> K[6. Attempt gpt-oss:20b] - K --> L{Loads in ~15 GiB?} - L -- no --> J - L -- yes --> M[Bench, record] - M --> N[7. Last: qwen3.6-moe:35b-a3b, service stopped] - N --> O[Record, decision] -``` - -1. Raise memlock. Add `* soft memlock unlimited` and `* hard memlock unlimited` to `/etc/security/limits.conf`. Then reboot. For a systemd unit, set `LimitMEMLOCK=infinity` instead. -2. Verify the NPU. Run `flm validate`. Run `xrt-smi examine`. Both must show the NPU. `flm validate` uses the kernel device. `flm run` uses XRT. A passing validate does not guarantee XRT sees the device. -3. Smoke test. Run `flm run qwen3.5:4b`. Toggle `/verbose` for performance reporting. Type `/bye` to exit. -4. Server mode. Run `flm serve qwen3.5:4b`. The default port is 52625. Use a scratch port if 52625 is taken. Never 7770. -5. Bench. Measure decode and prefill t/s at 1k/4k/16k starting context. Record to this page. -6. Step up in size: 9B, then gpt-oss-20b. -7. Attempt `qwen3.6-moe:35b-a3b` only after the smaller runs succeed, and only with the production service stopped. Watch `free -h` during the model download and load. - -## Registering packs absent from the catalog (NPU2) - -Use when adding a non-catalog NPU2 pack (an extraction or fine-tuned model) -to the FLM registry. Registry facts: user registry -`/home/dank/.config/flm/model_list.json`, shape -`{"model_path": "models", "models": {"": {"": {name, url, -file_url, size, flm_min_version, files[], vlm, default_context_length, -max_prefill_len, details{format:"NPU2", family, …}, label[]}}}}`; serve -resolves tags through `FLM_CONFIG_PATH` at startup, and `flm list` with the -env var set lists custom tags (`⏬` = not downloaded). - -1. Back up the registry JSON first (`cp -a`). -2. Insert the entry via a python json edit + atomic `os.replace`. Set - `files` to the FULL HF file list — **including `vision_weight.q4nx` even - for text-only packs**: the qwen3.5-family loader opens it unconditionally - and a missing file makes the first request HANG silently (no error log). -3. **Do not use `flm pull ` for custom entries** — it throws - `json.exception out_of_range key ':' not found` while building - the download list. Place files manually into - `/home/dank/.config/flm/models//` with - `curl -sfL "https://huggingface.co//resolve/main/?download=true"` - — plain `raw/main/` returns 135-byte LFS pointer stubs, not - weights; verify byte counts against the HF tree API before trusting a - download. -4. Kernel packs: copy from the same-shape family pack (shape-match - `config.json` on hidden_size/num_hidden_layers/head_dim first) into - `/opt/fastflowlm/share/flm/xclbins//` via `sudo cp -a` - (root-owned dir). -5. Deploy/restart the serve unit (the deploy path, not ad-hoc restarts) so - the new process loads with all files present. -6. Verify with a REAL completion request (`model: ":"`) — - `/v1/models` alone proves registration, not loadability. - -Hard constraints: only ONE `flm serve` process can hold the NPU hardware -context — a second concurrent serve process fails every load with -`DRM_IOCTL_AMDXDNA_CREATE_HWCTX err=-22` (aie2 mgmt msg opcode 0x2 status -0x2000003); never run scratch/verify instances while production holds the -NPU. Tag switching happens INSIDE one serve process; the first request -after a model switch pays ~6–11 s cold-load cost (visible in -`usage.load_duration`). Over ssh, never `pkill -f ""` with a -pattern present in your own command line — use bracket-guarded patterns -(`[q]wen3.5:4b`) or exact PIDs. - -## Tag load-failure diagnosis (200 with an error body) - -`POST /v1/chat/completions` answers **HTTP 200** with a flat -`{"error":"Failed to load model!"}` and no `choices`; `/v1/completions` -returns a nested OpenAI-shaped `server_error` (code 500) also under HTTP -200. A status-code-only client reads this as a successful empty completion — -**always check for the `error` key**. Two distinct causes, distinguished via -`journalctl -u --full`: - -1. **Unknown tag** (not in the registry): immediate error, nothing loaded. - Correct behavior, not a bug. -2. **Registered tag whose kernel pack is missing**: FLM resolves the tag, - downloads missing weight files on the fly (first request hangs ~90–110 s), - then fails `Failed to load model: No such file - '/home/dank/.config/flm/xclbins//.xclbin'`; subsequent - attempts fail fast (~2 s). Root cause: the serve script exports - `FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins` (user dir), so every - kernel lookup resolves under the user dir, which only contains symlinks - someone linked. **A tag listed in `/v1/models` is NOT proof it is - loadable on this host.** Fix: one symlink per missing pack - (`ln -s /opt/fastflowlm/share/flm/xclbins/ - /home/dank/.config/flm/xclbins/`) — verified live: after the - symlink the first request loaded and answered in 7.4 s wall. - -Switch-cost behavior: the `model` field routes per-request (no restart -needed); a request whose model differs from the resident one triggers a full -reload (7–11 s wall for an 8-token completion on ~4B-class packs); -alternating tags pays the swap each time. Single-lane NPU: concurrent -requests queue FIFO behind the current lock holder (`NPU Locked!` → -`NPU Lock Released!`). Probe recipe: `/v1/models` for registered tags; grep -the journal for `Failed to load model` (the quoted xclbin path names the -missing pack dir exactly); diff `ls ~/.config/flm/xclbins/` (symlinks = -installed) against `/opt/fastflowlm/share/flm/xclbins/` (available). - -## Incident: qwen3.6-moe unreachable (2026-08-16) - -Symptom: a client on dankware could not reach `qwen3.6-moe:35b-a3b` on port 52625. The journal shows three separate causes in the 14:19-15:03 window. - -1. **EXEC 203 crash loop (14:19-14:20).** The exec bit was lost on `serve-flm-small.sh` on the target. The unit failed `status=203/EXEC: Permission denied` four times, then started after the bit was restored. The git mode is `100755` (commit `3f42210`), and a `git pull` restores the bit. Verify with `git ls-files -s scripts/serve-flm-small.sh`. -2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-zealot` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-zealot' is not supported.` Each such request also triggered the unknown-tag fallback below. -3. **Swap churn (14:48-15:03).** The swap `gemma4-it:e4b` → `qwen3.6-moe:35b-a3b` restarted the service several times. Each load of the 23 GB model takes about 1.5-2.5 minutes at 262144 ctx. The endpoint does not serve during a load. - -Fix: the client model ID must be the FLM registry tag `qwen3.6-moe:35b-a3b`. The working pi agent entry in `~/.pi/agent/models.json` is `baseUrl http://192.168.0.143:52625/v1`, `id qwen3.6-moe:35b-a3b`, `contextWindow 262144`, `reasoning true`, `input [text, image]`. Verified end-to-end from dankware (192.168.0.226) at 15:32 on 2026-08-16. - -### Unknown-tag fallback (FLM behavior) - -When a request names a tag that is not in the FLM registry, FLM does not return a clean 404. It logs the ERROR, then falls back to the default model `llama3.2:1b`: - -- If the `llama3.2:1b` files are missing, FLM downloads them (1.2 GB on 2026-08-16) while the request hangs. -- FLM then loads the 1B model in place of the requested model. -- The files now exist in `~/.config/flm/models/Llama-3.2-1B-NPU2` (1.3 GB) on dankclaw. A future unknown-tag request loads the 1B model silently, with no download and no error. - -**Correction (live-probed 2026-08-30):** the fallback no longer fires on the -current serve path — an unknown tag now answers HTTP 200 with a flat -`{"error":"Failed to load model!"}` body and nothing is loaded or served; -`/v1/completions` answers a nested `server_error` (code 500) under HTTP 200. - -Keep client model IDs as FLM registry tags. Never send a service alias. Optional cleanup: move `~/.config/flm/models/Llama-3.2-1B-NPU2` (an incident artifact) to `.AGENTTRASH` so a stray tag re-downloads loudly instead of serving the wrong model. - -### Request compatibility: the `developer` role (found 2026-08-16) - -pi sessions failed against this endpoint with `Stream ended without finish_reason` while curl worked. Root cause, verified by field bisection: - -- The FLM Qwen chat template accepts only the roles `system`, `user`, and `assistant`. Any other role raises `Unexpected message role.` in the template. -- FLM reports the error differently per mode. Non-streaming: HTTP 500 with the template traceback. Streaming: HTTP 200, and the error JSON is the first body. A streaming client sees a stream with no chunks and no `finish_reason`. -- pi sends `role: "developer"` when `model.reasoning && compat.supportsDeveloperRole` (pi source, `openai-completions.js`). The default `supportsDeveloperRole` is true for standard OpenAI-compatible providers, so a reasoning model without a `compat` block always triggers the bug. -- Fix: `"compat": { "supportsDeveloperRole": false }` on the model entry in `~/.pi/agent/models.json`. pi then sends `role: "system"`. Verified with a headless pi session: full round trip, `finish_reason: stop`. -- FLM ignores unknown body fields. `reasoning_effort`, `store`, `max_completion_tokens`, and `tools: []` all arrive in pi requests and pass through without error. - -## Risks and Open Questions - -- **Ubuntu 26.04 is unofficial.** The distro list stops at 25.10. Validation passes, but breakage would fall on the "generic Linux" path. -- **The 30 GiB budget is untested at the very top.** gpt-oss-20b (15 GiB resident) works. 35B-A3B (~20 GB weights) is still an unknown. Vendor data covers 32 GB and 96 GB boxes only. -- **The FLM serve process was not under systemd** (2026-08-era risk). A reboot - of Dankclaw killed it. The fleet law today is the opposite: every deployed - container/service ships a boot-persistent, self-healing unit, and the retired - NPU unit is deliberately `masked` (`frank-npu.service`, read 2026-10-18) so it - cannot come back at boot. -- **gpt-oss reasoning budget.** The model can spend its full token budget on `reasoning_content` and return empty content. Clients must handle or cap this. -- **No llama.cpp GGUF path.** FLM uses its own pre-compiled kernels. Our UD-Q8_K_XL and HauhauCS builds are not importable. NPU work means FLM-specific model tags. -- **NPU production role (2026-08-16, retired 2026-09-11).** `frank-zealot` then - served `qwen3.6-moe:35b-a3b` on port 52625; the endpoint was retired with the - `amd_iommu=off` decision that day and the unit is masked today. The remaining - NPU content (benchmarks, runbook steps) is research on a device that is again - usable since 2026-10-18 but serves nothing. - -## Citations - -1. ROCm/FastFlowLM README (chip support, Linux support date, ROCm move, quick start, model storage paths): https://github.com/ROCm/FastFlowLM , accessed 2026-08-16. -2. FastFlowLM docs overview: https://fastflowlm.com/docs/ , accessed 2026-08-16. -3. FastFlowLM Qwen model cards (tags, quantization, context lengths): https://fastflowlm.com/docs/models/qwen/ , accessed 2026-08-16. -4. FastFlowLM gpt-oss model card (15.1 GB weight size, ~15.6 GB NPU cap, 32 GB inconsistency note): https://fastflowlm.com/docs/models/gpt-oss/ , accessed 2026-08-16. -5. FastFlowLM Qwen3.5 benchmarks (v0.9.38, Kraken Point 32 GB): https://fastflowlm.com/docs/benchmarks/qwen3.5_results/ , accessed 2026-08-16. -6. FastFlowLM Qwen3.6 benchmarks (v0.9.45, Kraken Point 96 GB): https://fastflowlm.com/docs/benchmarks/qwen3.6_results/ , accessed 2026-08-16. -7. AMD blog, "FastFlowLM Joins AMD to Advance AI Inference" (2026-07-17): https://www.amd.com/en/blogs/2026/fastflowlm-joins-amd-to-advance-ai-inference.html -8. Wikipedia, "Zen 5" article, Strix Halo section (Ryzen AI MAX+ 395: XDNA2 NPU, 50 TOPS, 256-bit LPDDR5X) via local ZIM `wikipedia_en_all_maxi_2026-02`. -9. Live system checks on Dankclaw, 2026-08-16: `flm validate` output, `dpkg -l` package list, kernel 7.0.0-29-generic, `/proc/3052096/fd` device fd inspection. diff --git a/serverAppConfigs-docs/LLM/FreeToken Builds.md b/serverAppConfigs-docs/LLM/FreeToken Builds.md deleted file mode 100644 index 87caf65..0000000 --- a/serverAppConfigs-docs/LLM/FreeToken Builds.md +++ /dev/null @@ -1,97 +0,0 @@ ---- -type: reference -title: FreeToken Builds -description: "Building the FreeToken runtime here: the CUDA and ROCm builds and the dependency trap each one hit." -tags: [freetoken, build, dankclaw, vllm, mtp] -timestamp: 2026-10-05T00:00:00-07:00 ---- - -# FreeToken Builds - -FreeToken = the official franksdank fork of FreeTok Tokenworks' freetoken -vLLM-MTP runtime. Two build paths exist, both on **Dankclaw**, with -different dependency traps. Both were verified end-to-end 2026-09-11/12. - -## Common verification (do not skip) - -**Verify the build by real completion, not by version string.** A version -bump alone does not move data through the model — a build can print a new -version and still not have completed a real token. Before declaring a -FreeToken build deployable, run an actual completion (any tiny prompt) and -read the returned content: - -```python -import json, urllib.request -req = urllib.request.Request("http://:/v1/chat/completions", - json.dumps({"model": "frank-zealot", "messages": [{"role": "user", - "content": "Hi"}]}).encode(), {"Content-Type": "application/json"}) -print(json.load(urllib.request.urlopen(req, timeout=600))) -``` - -A 200 with a real `choices[0].message.content` is the only build green. - -## CUDA build (the torch/vLLM dependency trap) - -Both paths build on **Dankclaw** with the same preinstall state: Python -3.12.10 in the system venv and **Torch 2.10.0.dev** (nightly, repo -download.pytorch.org/whl/nightly/cu130, installed with -`uv pip install --index-strategy unsafe-best-match "torch==2.10.0.dev+cu130" ---index-url https://download.pytorch.org/whl/nightly/cu130` — use -`--index-strategy unsafe-best-match`, `uv` otherwise refuses the dev -version). Verify the exact pair first; mismatched preinstalled deps are the -#1 build-failure cause. - -Then install the **matching vllm dev wheel**: - -```bash -uv pip install --index-strategy unsafe-best-match \ - --index-url https://wheels.vllm.ai//vllm-0.6.6.dev0+g.cu130-cp312-cp312-linux_x86_64.whl \ - --extra-index-url https://wheels.vllm.ai/nightly/cu130 -``` - -The wheel URL pattern is fixed: -`https://wheels.vllm.ai//vllm-0.6.6.dev0+g<8-char>.cu130-cp312-cp312-linux_x86_64.whl` -— read it back from vLLM's CI-run page (`run details → -build_and_install_package` job, "install the vllm wheel" step). **The trap: -vLLM refuses versions < its install_requires floor at runtime, so the wheel -version must be ≥ the runtime's floor.** A wheel built at a lower base -(0.5.x) downloads but `import vllm` fails with a version constraint error — -that failure looks like a build failure and is not one; upgrade the wheel, -not the code. - -If `uv pip install -e . --no-deps` refuses (some repos pin the -`--no-deps` branch off), fall back to `python3 setup.py` inside the venv, -after `python -m venv --copies .venv && source .venv/bin/activate` and -`uv pip install --no-build-isolation "torch==…"` — the venv must NOT -re-resolve torch (a second torch from PyPI d1 breaks the CUDA extension ABI). - -## ROCm build (gfx1151 via TheRock nightly) - -Same venv discipline, ROCm wheels from TheRock's nightly index: - -```bash -uv pip install --index-strategy unsafe-best-match \ - --index-url https://rocm.nightlies.amd.com/v2/gfx1151-dcgpu-supports_fbsgens_split_0717-rc1/ \ - "torch==2.10.0.dev+rocm" "triton==3.6.0" "pytorch-triton==3.6.0" -``` - -Then `vllm` itself (`uv pip install -e . --no-build-isolation --no-deps`, -repo branch `specs-mtp-dev`, commit `2ee440d`), after copying the two -prebuilt extension wheels into the tree (the repo's `setup.py` skips -compiling them when found). The ROCm build also needs -`install-fastrak-gfx1151.sh` from the repo's scripts and -`FASTRACK_HOME=/home/dank/FastTrack-runtime` in the unit env. FreeToken's -x86_64-gfx1151 prebuilt extension (`freetoken-ops`) ships on HF hub -`franksdank/freetoken-ops-gfx1151` — fetch it once and point -`FREETOKEN_OPS_HOME` at it; never rebuild the extension from source unless -the extension API changed. - -`--index-strategy unsafe-best-match` is load-bearing on both paths: the -nightly indexes carry version pairs that `uv`'s strict resolver otherwise -rejects, and the error (`No candidate found for pytorch-triton`) reads like -a missing package. - -## Related - -- [[Model Swap and Rollback]] — the swap/deploy story these builds serve -- [[Llama Cpp Serving Sizing]] — the harness measurement discipline diff --git a/serverAppConfigs-docs/LLM/GGUF Compatibility.md b/serverAppConfigs-docs/LLM/GGUF Compatibility.md index f66f42e..d0fd028 100644 --- a/serverAppConfigs-docs/LLM/GGUF Compatibility.md +++ b/serverAppConfigs-docs/LLM/GGUF Compatibility.md @@ -14,7 +14,7 @@ Studio refuse custom types at load (`unknown tensor type`); a fork-private type means exactly one engine can serve the file. Measured on this fleet repeatedly: `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is ftype 115 with 311 tensors of type 103 (`GGML_TYPE_Q8_0_ROCMFPX`) — only charlie12345/ROCmFPX reads it. The full type -tables live in [[Llama Cpp Runtimes]] and [[Llama Cpp Code Diff]] +tables live in and (ggml_type 100–108, ftype families 100–106 / 110–113 / 118). ## 1. Read the header, do not guess @@ -69,14 +69,14 @@ filenames lie about quantization and about MTP/vision heads. ``` Validate before trusting: the parse is right iff the tensor-info block ends - somewhere sane and names look real (`blk.0.attn_q.weight`, + somewhere sane and names look real (`blk.0.attn_q.weight` `token_embd.weight`, `output_norm.weight`). EOF throw = wrong enum or offset — do not work around it. 3. **Diff what engines actually bind on**: `general.architecture`; tensor count; tensor name list in order; per-tensor `ggml_type` and dims; geometry - KVs (`.block_count`, `.embedding_length`, `.feed_forward_length`, - `.attention.head_count[_kv]`, `.rope.*`, `.ssm.*`, `.context_length`, + KVs (`.block_count`, `.embedding_length`, `.feed_forward_length` + `.attention.head_count[_kv]`, `.rope.*`, `.ssm.*`, `.context_length` `.nextn_predict_layers`); MTP tensors (`nextn` in the name); vision tensors (`visual`, `v_` prefixes). A same-family fine-tune quantized by the same tool usually comes out byte-identical in every one of these — that is the @@ -117,8 +117,8 @@ re-download. ## Related -- [[Llama Cpp Runtimes]] — which fork ships which custom types -- [[Llama Cpp Code Diff]] — the enum tables and code-level differences -* [[kingjones-q8agent-rocmfpx]] — the era that served the ftype-115 file, and +- which fork ships which custom types +- the enum tables and code-level differences +* the era that served the ftype-115 file, and [[frank-dragoon]] for what runs there now - [[Llama Cpp Serving Sizing]] — pricing what a payload costs once it loads diff --git a/serverAppConfigs-docs/LLM/Heterogeneous Split Triage.md b/serverAppConfigs-docs/LLM/Heterogeneous Split Triage.md index 9952905..0f493b3 100644 --- a/serverAppConfigs-docs/LLM/Heterogeneous Split Triage.md +++ b/serverAppConfigs-docs/LLM/Heterogeneous Split Triage.md @@ -93,7 +93,7 @@ of bytes), so it is tempting. Before proposing it: (compare GGUF `tokenizer.ggml.tokens`, never trust the family name), or it must be a head **trained for that target** (EAGLE/DFlash/MTP are conditioned on the target's hidden states and are not portable between - models); + models) - a plain drafter costs **K sequential forwards per round**, so it must be 10–50× faster than the target. Break-even with `r = t_draft/t_target`: `rate_ratio = A/(K·r + 1)`, `A = (1−p^(K+1))/(1−p)`; at `r ≈ 0.4–0.5` @@ -107,10 +107,10 @@ of bytes), so it is tempting. Before proposing it: (draft gets its own device list) plus `-ngld`/`-otd`/`-ctkd -ctvd`. The llama.cpp half of that bullet is checkable upstream: `--spec-draft-device` -(`-devd` / `--device-draft`, line 4259), `--spec-draft-ngl` (`-ngld`, 4268), +(`-devd` / `--device-draft`, line 4259), `--spec-draft-ngl` (`-ngld`, 4268) `--spec-draft-override-tensor` (`-otd`, 4137) and `--spec-draft-type-k/v` (`-ctkd`/`-ctvd`, 4111/4124) all exist in -`common/arg.cpp` ([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/common/arg.cpp), +`common/arg.cpp` ([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/common/arg.cpp) read 2026-10-18). ## 6. Expressing a split (llama.cpp, the usual vehicle) @@ -122,8 +122,8 @@ read 2026-10-18). - buffer types match by name at load (`ggml_backend_buft_name`), so device names work as `-ot` values; -- `-cmoe`/`-ncmoe` are **CPU-only** (`--cpu-moe` / `--n-cpu-moe`, - `common/arg.cpp:2764,2771`); device-side MoE placement is `-ot`; +- `-cmoe`/`-ncmoe` are **CPU-only** (`--cpu-moe` / `--n-cpu-moe` + `common/arg.cpp:2764,2771`); device-side MoE placement is `-ot` - several backends in one binary is fine (CUDA + Vulkan is the practical pair; HIP + CUDA do not coexist *(unverified 2026-10-18 — upstream build docs do not settle the combination either way)*) — `GPU_BACKEND_A=OFF` in the @@ -155,7 +155,7 @@ Rank and report; prefer the boring answer when it wins: 1. **Two independent endpoints** (zero shared traffic) — often the real answer to "what's a useful combination", especially over a narrow link. 2. **One process, weights-resident, KV co-located with attention on the - fast pool** — promising and quantifiable (+% decode at long context); + fast pool** — promising and quantifiable (+% decode at long context) needs a dual-backend build and a baseline measurement. 3. **Hot-expert cache / per-token weight streaming** — affordable only if the miss set per token is small (a few experts; check @@ -169,5 +169,5 @@ Rank and report; prefer the boring answer when it wins: ## Related - [[PCIe Link and Atomics]] — the link and collective walls underneath -- [[Llama Cpp Multi-Device Splits]] — the deployed split shapes +- the deployed split shapes - [[Llama Cpp Serving Sizing]] — KV and pool arithmetic diff --git a/serverAppConfigs-docs/LLM/Hybrid CPU GPU Processing Dankserver.md b/serverAppConfigs-docs/LLM/Hybrid CPU GPU Processing Dankserver.md deleted file mode 100644 index 017109f..0000000 --- a/serverAppConfigs-docs/LLM/Hybrid CPU GPU Processing Dankserver.md +++ /dev/null @@ -1,225 +0,0 @@ ---- -type: reference -description: "Dankserver uses NVIDIA GPU with system RAM for hybrid processing." -tags: - - llm -timestamp: 2026-08-28 ---- - -# Hybrid CPU GPU Processing Dankserver - -## Overview - -Dankserver has **no NVIDIA GPU**: the RTX 3090 left the box on 2026-09-15 and the host now runs two AMD Radeon AI PRO R9700s (`lspci | grep -ci nvidia` → 0, read 2026-10-18 — [[Dankserver System Specs]]). This page's vLLM-era hybrid guidance is reference material, not the serving path: the live service is radiance 1.0.13 as `frank-archon`, and its hybrid mechanism is a pinned host pool (43.88 GiB per rank, `scripts/frank-archon.env:184`) plus the CPU-resident 47.68 GiB n-gram/PLE table, not vLLM KV swapping ([[frank-archon]]). - -## Architecture - -### Model Weights - -In the setup this page describes, model weights stay on GPU and the GPU must hold all parameters. As a statement about vLLM it is no longer true: current vLLM ships CPU weight offload as `cpu_offload_gb`, plus the grouped prefetch path (`offload_group_size`) — `vllm/config/offload.py:23,54` ([source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/config/offload.py), read 2026-10-18). What remains true is the capacity arithmetic: offloaded weights cost PCIe traffic per token (see [[PCIe Link and Atomics]] §2). - -### KV Cache - -KV cache moves between GPU and CPU: - -1. Active KV cache pages stay in GPU VRAM -2. Inactive pages swap to CPU RAM -3. Swapped pages write to disk swap space if RAM is full -4. Pages reload to GPU when needed - -### Compute Distribution - -- GPU handles all matrix multiplication and attention computation -- CPU handles tokenization, detokenization, and input processing -- CPU handles KV cache swap operations -- GPU worker processes run on CPU cores via multiprocessing - -## Optimal Settings for Dankserver - -### GPU Memory Utilization - -Set `--gpu-memory-utilization` based on your GPU: - -| GPU | Recommended Value | -|-----|-------------------| -| 24 GB (RTX 3090/4090) | 0.7-0.8 | -| 40 GB (A100) | 0.8-0.9 | -| 80 GB (A100 80GB) | 0.85-0.95 | -| 80 GB (H100) | 0.9-0.95 | - -Dankserver has neither NVIDIA card today: it carries 2× 32 GB (32,624 MiB) R9700s. -The live engine does not use this knob at all — it claims an explicit per-card -pair instead (`--vram-weights-mib 25600` / `--vram-kv-mib 5120`, -`scripts/frank-archon.env:230-231`). - -Lower values leave more GPU memory for KV cache. This reduces swap frequency. - -### Swap Space / KV Offload - -Current vLLM has no `--swap-space` engine argument, and the CPU KV-offload knob pair -is owned by [[RAM CPU Offloading in VLLM]] (that page carries the upstream -`CacheConfig` verification). In Podman, mount the host cache read-write for KV offload -writes. The table below shows historical `--swap-space` guidance as a sizing reference. - -| System RAM | Recommended Swap | -|------------|------------------| -| 32 GB | 8-16 GB | -| 64 GB | 16-32 GB | -| 128 GB | 32-64 GB | -| 256 GB | 64-128 GB | - -More swap space allows longer contexts and more concurrent requests. - -### Tensor Parallelism - -Use tensor parallelism for large models: - -``` -vllm serve meta-llama/Llama-3.1-70B-Instruct \ - --tensor-parallel-size 4 \ - --gpu-memory-utilization 0.85 -``` - -### KV Cache Data Type - -Use FP8 KV cache for Ampere and newer NVIDIA GPUs. On this host's RDNA4 cards the live engine runs an fp8 KV cache too (`kv fp8` — `scripts/frank-archon.env:264`, which also records that this engine takes only bf16|fp8 for the whole cache). - -``` -vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \ - --kv-cache-dtype fp8 \ - --gpu-memory-utilization 0.9 -``` - -FP8 KV cache reduces memory usage by 50% compared to BF16. - -## NUMA Optimization - -Dankserver reports **one** NUMA node today (`lscpu` → `NUMA node(s): 1`, `NUMA node0 CPU(s): 0-7`, read 2026-10-18), so NUMA binding buys nothing on this host; these flags matter on multi-node topologies: - -``` -vllm serve model \ - --tensor-parallel-size 4 \ - --numa-bind -``` - -This pins each GPU worker to its nearest NUMA node. This reduces memory access latency. - -For explicit NUMA binding: - -``` -vllm serve model \ - --tensor-parallel-size 4 \ - --numa-bind \ - --numa-bind-nodes 0 0 1 1 -``` - -## CPU Resource Requirements - -VLLM needs CPU cores for: - -- 1 API server process -- 1 engine core process -- N GPU worker processes (one per GPU) - -Minimum physical cores = 2 + N (where N is GPU count). - -For Dankserver, which has **2** GPUs (2× R9700) and 8 cores (`lscpu`, read -2026-10-18): - -- Minimum 4 physical CPU cores by the 2 + N rule -- Recommended 8+ physical CPU cores — the host has exactly 8 (SMT off) - -## Performance Tuning - -### Chunked Prefill - -Chunked prefill is enabled by default. Tune with: - -``` -vllm serve model \ - --max-num-batched-tokens 8192 -``` - -Higher values improve throughput. Lower values improve latency. - -### CUDA Graphs - -CUDA graphs improve throughput. Disable for faster startup: - -``` -vllm serve model \ - --enforce-eager -``` - -### Compilation Cache - -Reuse torch.compile artifacts: - -``` -export VLLM_CACHE_ROOT=~/.cache/vllm -vllm serve model -``` - -Copy cache between boots for faster startup. - -### Skip Memory Profiling - -Skip memory profiling on repeated boots: - -``` -vllm serve model \ - --kv-cache-memory -``` - -## Monitoring - -Check GPU memory: - -``` -rocm-smi -``` - -Check for preemption warnings in VLLM logs. - -Check CPU utilization: - -``` -htop -``` - -## Example Commands - -### Single GPU, Medium Model - -``` -vllm serve Qwen/Qwen3.6-35B-A3B \ - --gpu-memory-utilization 0.85 \ - --max-num-batched-tokens 4096 \ - --max-num-seqs 128 -``` - -### Single GPU, FP8 Model - -``` -vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \ - --kv-cache-dtype fp8 \ - --gpu-memory-utilization 0.9 \ - --max-num-batched-tokens 8192 -``` - -### Multi-GPU, Large Model - -``` -vllm serve meta-llama/Llama-3.1-70B-Instruct \ - --tensor-parallel-size 4 \ - --gpu-memory-utilization 0.9 \ - --max-model-len 8192 -``` - -## See Also - -- [[VLLM Overview]] -- [[VLLM Index]] -- [[RAM CPU Offloading in VLLM]] -- [[Shared Memory Architecture Dankclaw]] -- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] diff --git a/serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md b/serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md deleted file mode 100644 index 1668a50..0000000 --- a/serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md +++ /dev/null @@ -1,271 +0,0 @@ ---- -type: reference -title: LLM Bench Windows and Measurement -description: "Opening and restoring benchmark windows on fleet hosts, honest methodology, and the usage-counter law under drafting." -tags: [benchmarking, llama-bench, ab-testing, speculative, vllm, fleet] -timestamp: 2026-10-05T00:00:00-07:00 ---- - -# LLM Bench Windows and Measurement - -Fleet law first: **STOP the production service before any GPU experiment** -(AGENTS.md Benchmarking Policy) — a bench beside a live service starves the -GPU and corrupts results; an unisolated GPU experiment has precedent for -wedging a host until power cycle. Reference harnesses: -`scripts/bench-batch-ubatch.sh`, `scripts/autoresearch.sh`, -`scripts/autoresearch/frank-archon-arms.sh`. - -## 1. The window structure (one config per arm, a driver owns the window) - -1. **Stop + gate before every arm** — `wait_for_clear`: unit NOT active AND - no RUNNING container of that name (`podman ps`, not `-a`; `-a` counts - EXITED containers and can never clear — `rm -f` the exited one first) AND - port free (`ss -ltn`). The graceful-stop race cost a whole arm 2026-10-01: - `ExecStop=podman stop -t 120` drains up to 2 min; a driver that sleeps 3 s - then polls `/health` gets **200 from the still-running PRODUCTION engine** - — the arm "passes" with production's boot log, then `podman exec` dies - with `container state improper`. -2. **Boot the arm as the service, not a scratch port** (no scratch ports on - LLM hosts — one card, one server). The serve script takes its config via - env-file override (`FRANK_ZEALOT_ENV_FILE`, `FRANK_ARCHON_AR_ENV_FILE`) - pointing at a staged copy; the checkout is never mutated. -3. **Fast-fail the health wait**: poll `podman ps | grep -qx ` inside - the loop — a refused boot dies in one poll instead of 20 min. Capture the - serve script's FULL stdout on failure (a `tail -8` loses engine-init root - causes; raise happens above the tail). -4. **Measure** (§4), capture boot-log engine facts (KV pool, arena slots, - SpecDecoding lines) — they survive measure crashes. -5. **Teardown + converge**: kill by explicit PID/pgid (never pattern kills), - `podman rm -f` scratch, wait `MemAvailable ≥ ~230 GiB` on dankserver - before the next boot; restore production from the EXIT trap and verify the - **conjunction oracle**: health 200 + `is-active` + `is-enabled` + - mask-symlink clean. A bare `systemctl start` is not a restore oracle. - (`systemctl mask` FAILS on a relinked unit — log it, explicit stop is - enough: systemd never auto-restarts an explicitly stopped unit.) -6. **Warm boots cycle ~2 min** (page cache), cold ~8 min — size timeouts for - mixed speeds. - -**Destructive tails (2026-10-01):** a scratch script whose tail does -`systemctl stop` + `podman rm -f ` fires later if a -probe loop survived the driver kill — it can destroy a just-deployed -production engine (signature: deploy exits green, then seconds-to-minutes -later `Exited (0)`, `Deactivated successfully`, clean exit = removed, not -crashed). Before ANY deploy: `pgrep -af` every scratch script of the session -and kill strays by explicit PID; killing the dev-side job does NOT kill the -host-side `deploy-remote.sh` (two of them once managed the same unit). - -Two failure shapes inside the window that read as "the bench died": - -- **Stale reads from a stopping container.** `podman stop` answers while the - engine inside is still flushing; a `podman logs` taken in that window - returns the boot-prefix only. Poll `podman ps` for `Exited (0)` (or - `podman wait`) before reading logs, or you will write "no results" into - the table for a run that has them. -- **A D-state worker wedges the stop.** A bench child that wedged in the - kernel ignores SIGTERM, so `podman stop -t N` burns its full timeout and - then the container is `Stopping` forever. Kill by container first - (`podman kill ` → SIGKILL path), verify the process is gone from - `podman ps` AND `nvidia-smi`, and only then re-serve. Never let the - window script proceed to a redeploy with a zombie bench worker holding - VRAM. - -## 2. llama-bench sweeps (offline batch/ubatch/split/KV/depth) - -**What llama-bench cannot express — decide BEFORE designing:** no -`--ctx-size` (`n_ctx = n_prompt + n_gen + n_depth`, so a bench can NEVER prove -activation buffers fit beside the DEPLOYED KV — only the real unit answers -that), no `--parallel` (no slot shape, no concurrency), no `--spec-*` -(`tg` numbers are the drafterless baseline, useful as a control only), and a -warmup run is itself a full-size run. `-d N` is a depth PREFIX (paid once per -cell via a `llama_state_seq_*` save/restore); without it a `tg` run is at -~zero depth and meaningless for long conversations. - -**The separator trap:** `/` separates devices WITHIN one configuration, `,` -separates sweep values — for both `-dev` and `-ts`. `-ts 36,29` parses as the -single value 36 → everything on one card. `--tensor-split 36,29` (server) → -`-ts 36/29` (bench). Ranges exist: `-ub 512-2048+256`. - -**Harness shape:** one invocation per cell (an infeasible cell calls -`exit(1)` and kills every later cell of a cross-product invocation — and -infeasibility is usually the question); TSV row per cell as it completes + -`bench.json` (`-o json`) + `bench.log` (`--progress`) + a 2 s `rocm-smi ---showmeminfo vram` VRAM trace per cell (a cell that allocated less than -intended looks fast); resume by skipping non-empty cell outputs; stop the -service in the harness, restore from `trap … EXIT`, verify afterwards. Match -the deployment's load flags exactly (`-ngl -sm -ts -fa -ctk/-ctv -t`). - -**Run under a transient unit, not a shell:** - -```bash -sudo systemd-run --unit= --collect --no-block --property=Type=oneshot \ - --property=LimitMEMLOCK=infinity --property=User=dank \ - /home/dank/serverAppConfigs/scripts/.sh -``` - -Deploy units set `LimitMEMLOCK=infinity` for ROCm staging; an SSH shell has -8 MiB and fails the load for an unrelated reason. This also survives ssh -disconnects. **Boot rungs under the production environment, not a bench -shell** — a rootless shell probe booted 3×262144 bf16 that production refused -(systemd `User=` + memlock change what the pinned-expert path holds): -`sudo systemd-run --uid= --unit= -p LimitMEMLOCK=infinity -p -Restart=no env VARFILE=… /path/serve.sh`. - -**Verify before burning hours:** dry-run against a STUB binary with -`SERVICE=` (the trap/stop never reached), tiny grid, -including one forced-`exit 3` cell — prove the TSV row, failure capture, -resume logic. Validate the real flag set in ~1 s by pointing `-m` at a -nonexistent file. **Reading:** bracket the sweep (reference cell first AND -last); check per-cell `stddev_ts` (a difference smaller than spread is not a -result); single-sequence decode is usually flat across batch/ubatch (prefill -knob); batch ≥ ubatch (llama.cpp clamps ubatch to batch, silently duplicating -smaller-batch cells); report what the grid cannot prove (drafter, slots, real -context). - -## 3. HTTP benches of server-side parameters - -`llama-bench` benches the bench binary, not the running server — server-side -params (`--threads-batch`, `--ubatch-size`, `--spec-type`, `--parallel`) need -HTTP: - -- **Prefill**: large prompt, `max_tokens=1` (server crashes on 0), measure - `prompt_tokens / wall_time`. -- **Decode**: `max_tokens=16384` minimum, NEVER 1 (warmup, cache growth and - spec acceptance all stabilize over thousands of tokens); measure - `predicted_per_second` from `timings` or `completion_tokens / wall_time`; - temperature 0. - -Known fleet harnesses: `scripts/bench-threads-batch-http.py`, -`scripts/bench-decode-threads-batch.py`, `scripts/bench-spec-decode.py`. -sglang-specific: `max_tokens=1` on `/v1/chat/completions` with the qwen3 -reasoning parser streams zero deltas — use `/v1/completions`. - -## 4. The measurement law (under speculative decoding this is absolute) - -**Never count SSE chunks.** Rejected drafts break the 1:1 chunk↔token -mapping; chunk counting reported 8.3 tok/s for a config whose server counters -said 23–26 — a 3x false regression that nearly abandoned the change that -worked. Always: - -```python -decode_tok_s = usage.completion_tokens / (wall - ttft) -# stream=True, stream_options: {include_usage: true}; ttft = first content-bearing chunk -``` - -vLLM's 10 s `Accepted throughput` log lines are a rolling window, not a -measurement (6.2 → 13.1 across adjacent windows on identical settings) — use -them for acceptance STATISTICS only (`Avg Draft acceptance rate`, `Mean -acceptance length`, per-position rates). Warm pass + ≥3 runs, median; -temperature 0; ≥256 tokens. **A decode "jump" with early-stopped completions -(ct < 256) and acceptance > 90 % is degenerate generation** (the fork's -repetition-halt fired at ~140 tokens on corrupted numerics) — inspect text -before believing a win. Run-to-run spread is 5–10 %: repeat arms; one sample -nearly shipped the wrong setting. - -## 5. Honest methodology - -- **Match the corpus to the workload.** Token soup under-reports speculative - decoding (2.9–3.6 tok/update vs 3.86–5.8 on real code) and mis-ranks draft - settings; acceptance, not step time, moves throughput. Build the corpus - from the repo's own sources: `{"prompt": "\n\n"}`, - `--custom-output-len`. -- **Salt every run** with a unique per-arm token in the first bytes — it is - the prefix-cache key (and a proxy routing key). Without it arm 2 measures - arm 1's warm cache. -- **Two workload shapes need two clients:** distinct prompts (batch - behaviour, prefill parallelism) and sequential turns with growing history - (TTFT of the first streamed chunk per turn — the only measurement that - exposes prefix-cache behaviour). These pull in opposite directions: a cold - batch spreads prefill across replicas, a conversation's next turn only pays - if it lands where its prefix is cached. -- **Read the engine's ledger, not the card:** `Available KV cache memory` / - `GPU KV cache size: N tokens` / `Maximum concurrency for M tokens per - request: X.Xx` is the capacity number. vLLM's `Prefix cache hit rate:` line - is a LIFETIME average across every prompt the replica ever saw — send the - same prompt twice to an idle engine and compare TTFTs (measured 22.3 s cold - → 1.4 s warm for a 50k prefix). `GPU KV cache usage` says the pool is not - full; it does NOT say a prefix survived. The KV budget is NOT monotonic in - chunk size (5.44 GiB at CHUNK 8192, 4.22 at 16384, 4.16 at 4096) — measure - three points before treating a knob as a lever. -- **Soak before believing:** 15–30 min of concurrent families, count failures - AND latency; report hit/miss split and medians with min/max — the median of - a bimodal distribution describes neither mode. -- **Verification that is not the benchmark:** run the client the fleet - actually uses; and a deployed unit is not proven until an unattended reboot - brings it back. -- **Record the "no" answers** with numbers and mechanism — placement balance, - concurrency admission and chunk-size tweaks all measured flat or worse - here; the wall was arithmetic (cached prefixes + live sequences share one - pool). - -## 6. Sweep hygiene (the arms that produced wrong verdicts) - -- **Prefetch uniformity:** every arm in a sweep carries the same - expert-prefetch setting. The 2026-09-29 campaign measured losing arms with - prefetch ON and incumbents OFF; untaxed, verdicts inverted (no-spec - 12.11 → 41.47; gather 32×64 parity → +18.7 %). Never re-read a taxed - verdict. -- **Knob-landed preflight:** for every `-e` knob, confirm - `podman exec printenv ` — a knob the fork silently ignores - produces a "parity" arm that is really the default (v1 of the gather sweep - lost all 3 arms this way; serve-script hooks are additive and can drop - vars). -- **Always run an explicit OFF arm:** the 2026-09-29 campaign's only win was - deleting a baseline's silent `--expert-prefetch 1` (+40–51 %). Challengers - compare against default-off, not "incumbent + one flag". -- **Pin the capacity bar:** `--kv-cache-memory-bytes` (contract) rather than - autofit (remainder) so pool numbers are reproducible; compute need = - tokens × measured B/token (hybrid models: KV dtype does NOT double mamba - states — measure, don't assume 2x). -- **Report decode AND prefill (TTFT) per rung:** a decode-only view nearly - deployed a 2x-prefill-tax config. Per-depth tables, never a single median — - one fork's gather win grows with depth (54 t/s @1k, 39.5 @short). -- Container recreate (`podman rm -f`) wipes `podman cp`'d harnesses — re-copy - `mtp_measure.py`/`bench_iter.py` after EVERY recreate, or mount them. - -## 7. Bash laws that ate windows (each cost an arm) - -1. `${VAR:+SPEC=$1}` in front of a command expands to a bare word → - `SPEC=3: command not found`, nothing runs. Pass literal assignments: - `SPEC="${1:-}" bash serve.sh`; the callee's `${SPEC:+--flag}` skips empty. -2. `${2WAY:-0}` is fatal `bad substitution` (digit-leading name), and inside - an if/else it aborts ONLY that compound statement — the script CONTINUES, - every arm tears down clean with zero measurements. Detection law: `ls` - the per-arm `.measure.txt` files before trusting any summary; arm echoes - are not evidence. -3. `local a=$1 b=$x` expands `$x` before the builtin assigns — under `set -u` - it aborts AFTER the harness stopped production. Two statements. -4. `sed -i` renames the file; already-running bash keeps the OLD inode — - mid-flight patches miss arms launched in the preceding minutes. Patch - early or run a make-up window. -5. Never heredoc python through ssh→zsh→bash — write to a file, ship it - (`ssh host 'cat > /path && chmod +x /path' < local.sh`); nested quoting - fails silently. Same trap in `git commit -m`: backticks become command - substitution. -6. No comments INSIDE a backslash-continued command (the continuation joins - the comment); verify multi-line command construction with `bash -n` AND a - render test. - -## 8. Results location and recovery (frank-archon arms) - -| copy | path | -|---|---| -| per-arm evidence | `$ARMS_DIR` = `~/.jobs//` (`.rc`, `.done` beside it) | -| summary | `$ARMS_DIR/summary.tsv` = `arm rc decode_tps decode_tps_t0.7 kv_tokens boot_s` + `.out/.env/.overlay/.rc/.metrics` | -| server-side last window | `/home/dank/.autoresearch-frank-archon/run.json` | -| durable memo | `.../last/run.json` + `last/hash` (promoted only after bench AND verified restore) | -| fallback | `~/.omp/agent/sessions///.{shake,bash}.log` (permanent) | - -`/tmp/frank-archon-arms-*` and `/tmp/armstub/` are stub self-tests — -constants 99.9/88.8 identical across arms by construction; **identical -numbers across arms == stub or bug, never a measurement.** After compaction, -re-harvest with `grep -rhoE 'decode_tps[a-z0-9_.]*=[0-9.]+'` over the session -logs. A window costs ~20 min warm; a detached runner's timeout ≥ 3600 s per -window. - -## Related - -- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan arithmetic being swept -- [[DFlash DSpark Speculative Decoding]] — spec A/B protocol and shapes -- [[LLM Performance Diagnosis]] — profiling and attribution methods -- [[Remote Job Runner]] — detached execution of long sweeps -- [[frank-medium-llamacpp-dflash2]] — the measured ubatch-optimum history diff --git a/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md b/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md index 7ccd078..9a493df 100644 --- a/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md +++ b/serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md @@ -21,7 +21,7 @@ the cost; the problem is per-token expert movement and KV/attention tuning is ruled out entirely. A genuinely context-bound server shows a clear downward slope. Record cold vs warm separately — a small gap means the cache is working. The canonical flat run (fp8 expert-offload shape, a few hundred to -sixteen thousand tokens of context) is in [[Tcclaviger Fork Ops]]'s profile. +sixteen thousand tokens of context) is in 's profile. ## 2. Duty-cycle arithmetic (the "GPU not fully utilized" answer) @@ -86,7 +86,7 @@ Worked example (GLM-5.3-Flash, 2×GPU, offload, decode 9.9 tok/s): No amount of link arithmetic would have produced those two conclusions: the suspect was 1 %, and the mass was pointwise glue from an uncompiled trunk — -whose own boot log names the cause (`FULL_DECODE_ONLY` override). It also +whose own boot log names the cause (`FULL_DECODE_ONLY` override). It predicted a negative result in advance (spec decode cannot help a shape whose per-token cost scales with the token count) which matched an unexplained flat MTP measurement. @@ -117,7 +117,7 @@ per-request telemetry — it measures the real workload and costs nothing. - One `max_tokens=200` request is production traffic and fine; anything that pins the card is a bench → stop-and-mask first. Check WHICH instance you are reading (`systemctl status | head -12`; task ids jump across - restarts). The engine's own ledger (`KV self size`, `n_slots`, + restarts). The engine's own ledger (`KV self size`, `n_slots` `n_ctx_slot`) outranks your model of it. ## 5. The fork's data path (before blaming the link) @@ -130,7 +130,7 @@ tokens, so the step can silently cross the threshold onto the slow path as draft depth grows. Check the threshold's premise: a default tuned at 12.4 MiB/expert does not transfer to 2.344 MiB/expert, where copy-in can be several times faster. TunableOp ghost-skip: `CLAV_TUNABLEOP_SWEEP=1` alone -does NOTHING under expert offload (`CLAV_TUNABLEOP_WITH_OFFLOAD=1` required); +does NOTHING under expert offload (`CLAV_TUNABLEOP_WITH_OFFLOAD=1` required) check for the skip line before concluding autotuning ran. Gather-geometry knobs read at import time → each arm is a full reload. @@ -189,7 +189,7 @@ analyze; `--analyze-only`). Measure with a **sequential-turn client**, not only a batch benchmark — a batch of distinct prompts parallelises prefill across replicas and hides the problem. The two failure modes: round-robin loses every other cache hit -(90k-token contexts: warm-turn TTFT 2676 ms round-robin vs 1096 ms pinned); +(90k-token contexts: warm-turn TTFT 2676 ms round-robin vs 1096 ms pinned) pinning-everything loses a card (16 requests: 18.6 s spread vs 30.1 s pinned). **The rule that gets both: route by request shape** — a request carrying more than one message is a continuation (agent loop re-sending its @@ -209,13 +209,13 @@ is not a per-request verdict (probe directly: same prompt twice to an idle replica — 22.3 s cold → 1.4 s warm); salt corpora; record sample sizes; `vllm bench serve` needs `--tokenizer `; verify metric names against the live endpoint before dashboards. Test shape: drive the routing -function directly with synthetic bodies (stdlib-only policy module); +function directly with synthetic bodies (stdlib-only policy module) conversation identity is the FIRST N KiB of prompt — a shorter tested prefix hashes differently per turn and pinning silently never happens. ## Related -- [[LLM Bench Windows and Measurement]] — the window discipline and usage-counter law +- the window discipline and usage-counter law - [[PCIe Link and Atomics]] — link probing and the streaming-vs-compute ceiling - [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan behind duty-cycle terms -- [[vLLM Porting and Compile Triage]] — the compile fix for the pointwise-glue bound +- the compile fix for the pointwise-glue bound diff --git a/serverAppConfigs-docs/LLM/Llama Cpp Multi-Device Splits.md b/serverAppConfigs-docs/LLM/Llama Cpp Multi-Device Splits.md deleted file mode 100644 index 896f0ab..0000000 --- a/serverAppConfigs-docs/LLM/Llama Cpp Multi-Device Splits.md +++ /dev/null @@ -1,287 +0,0 @@ ---- -type: reference -title: Llama Cpp Multi-Device Splits -description: "What llama.cpp can express across devices: the no-collective layer-split path, balancing, and drafter KV sizing." -tags: [llama.cpp, multi-gpu, layer-split, pcie, sizing, speculative] -timestamp: 2026-10-05T00:00:00-07:00 ---- - -# Llama Cpp Multi-Device Splits - -## 1. Expressibility law — classify the arrangement before running anything - -llama.cpp expresses exactly two things across devices, and they are not -interchangeable: - -| arrangement | how | expressible? | -|---|---|---| -| **Layer split** — layers 0..n on dev A, rest on dev B | `-sm layer -ts /` | **yes**, no collectives needed; works cross-vendor (CUDA + Vulkan measured 2026-09-16) | -| **Per-tensor placement** — attention+KV on A, experts on B | `-ot =` | **NO.** `--override-tensor` sets `has_tensor_overrides()`, a REQUIRED term of the scheduler's `pipeline_parallel` flag (only read site: `llama-context.cpp`, in the `pipeline_parallel = …` expression). That flag sets `n_copies = parallel ? MAX : 1` in `ggml_backend_sched_new`; with `n_copies == 1` a pre-allocated weight on a non-primary device has no backend and the graph aborts at `ggml-backend.cpp:941` (`pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the operation (NONE)` — `(NONE)` is the leaf's own op name). | - -Upstream anchors, re-read 2026-10-18 against `master`: -`src/llama-context.cpp:517-526` computes `pipeline_parallel` with -`!model.has_tensor_overrides()` as a required term; -`ggml/src/ggml-backend.cpp:1089` carries the verbatim abort -`pre-allocated tensor (%s) in a buffer (%s) that cannot run the operation (%s)` -(the `:941` anchor above is where it sat in the `37b53fd` build this was -measured on); and `common/arg.cpp` defines `-sm/--split-mode` (2821), -`-fit/--fit` (2882), `--spec-type` (4295), `--spec-draft-device` (4259), -`--spec-draft-n-max` (4161), `-fa/--flash-attn` (1758) — every flag in §4 is -real upstream, not fork-only -([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/common/arg.cpp)). - -Failure modes observed (mainline `37b53fd`): per-tensor `-ot exps=Vulkan1` -with `-dev CUDA0,Vulkan1` → scheduler abort; layer-scoped `-ot` with a -cross-backend pair → model load fails with only `failed to load model`; -`-ot x=CPU` **works** because the CPU backend is last in the list — **the CPU -is the only legitimate partner in a heterogeneous split.** On a CUDA+AMD host -a card can be the trunk OR the offload host, never the complement of an -AMD-resident peer. Pure Vulkan changes nothing here — the gate is the -override, not the driver mix. - -If the plan needs tensors on a specific device: stop, the answer is no for -any vendor mix. Expressible alternatives, in order: (1) two independent -endpoints, one model per device; (2) single-vendor layer split; (3) device + -CPU per-tensor offload (mainline pinning guard: `GGML_CUDA_NO_PINNED`; ik fork -was `GGML_CUDA_NO_PINNED_WEIGHTS` — without it a CPU-overridden tensor can -land in a pinned `CUDA_Host` buffer and page-lock ~108 GiB). - -Layer-split arithmetic to check before promising: a whole layer carries its -experts (MoE) — compute GiB/layer, not per param-group; only some layers carry -KV (every 4th on the hybrid), so a share of layers is only a share of KV; big -non-repeating tensors (PLE tables) land on the main device and cannot move — -they cap how small the primary can get. - -## 2. No collectives? The atomics wall and the layer-split answer - -Decide with one line before designing: - -```bash -sudo lspci -vv -s | grep AtomicOpsCtl # ReqEn- ⇒ no collectives, ever -lspci -tv # a Routing- bridge is a hardware wall -``` - -`ReqEn-` kills vLLM entirely — TP needs a per-layer allreduce AND PP needs a -device-backend `send`/`recv` of the hidden state -(`vllm/distributed/parallel_state.py`) — i.e. the same RCCL primitive that -fails on an atomics-less card. llama.cpp's `--split-mode layer` needs no -collective: layers distribute in order, only the boundary activation crosses -PCIe once per forward pass. Full probe/repair detail: -[[PCIe Link and Atomics]]. - -## 3. Build: pin by feature, bundle the runtime at the exact depth - -- Pin the ref by FEATURE list (e.g. DFlash2 + qwen35 MTP on 2026-09-16 → - `ggml-org/llama.cpp#27342`); never float. Verify a fork build the same way - (`ls /…/spec_decode/ # dflash/ vs dflash2/`). -- `-DCMAKE_HIP_ARCHITECTURES=` — a sibling arch links, loads, then - segfaults. `-DGGML_HIP_RCCL=OFF` when there is no atomics path (only tensor - mode degrades to the butterfly fallback; layer split unaffected). -- **TRAP — `cp -r SRC DST/` flattens when `DST` does not exist** and drops one - directory level. rocBLAS/hipBLASLt look for device code at a FIXED depth - (`../lib/rocblas/library`, `../lib/hipblaslt/library/`); flattened, - the engine loads every GiB, answers `/health`, and dies on the FIRST request - with `Cannot read .../TensileLibrary.dat: No such file or directory for GPU - arch`. `mkdir -p DST && cp -r SRC/. DST/`, then - `patchelf --set-rpath '$ORIGIN/../lib' build/bin/llama-*`. -- Dual-backend build (CUDA + Vulkan is the practical pair; HIP+CUDA do not - coexist): `-DGGML_CUDA=ON -DGGML_VULKAN=ON` in one tree; Vulkan needs - `glslc` + libvulkan headers. **Device names are not what you expect:** - `--list-devices` lists the discrete card under BOTH backends (`CUDA0` AND - `Vulkan0`) and the APU LAST (`Vulkan1`) — a wrong `-ot` name makes - llama.cpp print the valid set and exit (self-diagnosing). -- What the existing build can do: `GPU_BACKEND_*=OFF` lines in its - CMakeCache. - -## 4. Serve flags that matter on a split - -```bash -HIP_VISIBLE_DEVICES=0,1 llama-server \ - --device ROCm0,ROCm1 --split-mode layer --tensor-split 1,1 \ - --gpu-layers 999 --fit off --flash-attn on \ - --ctx-size --parallel 1 --cache-type-k f16 --cache-type-v f16 \ - --jinja --metrics --cache-prompt \ - --spec-type draft-dflash --spec-draft-model \ - --spec-draft-device ROCm1 --spec-draft-n-max 7 -``` - -- `--fit off` + explicit `--ctx-size`: auto-fit must not resize a context set - on purpose. **`--parallel N` DIVIDES `--ctx-size`** — one slot at large ctx - is the concurrency policy, concurrent agent loops queue. -- Pin the drafter to the card holding the target's LAST layers - (`--spec-draft-device`); under tensor mode an unpinned draft joins the - meta-device KV allocation and aborts the KV build. `--spec-draft-n-max` must - be ≤ the drafter's `block_size - 1` (DFlash2: 8 ⇒ 7). -- Qwen3.8-family templates RAISE on `reasoning_effort=high` (500) — probe - low/medium/high/xhigh and map harness levels before declaring ready. - -## 5. Split balancing: measure, never interpolate - -**A `-ts` step is ONE LAYER, and hybrid stacks do not have uniform layers.** -qwen35 is 16 × (3 Gated DeltaNet + 1 Gated Attention): one step ≈ 0.43 GiB on -a DeltaNet boundary, ≈ 1.43 GiB on a full-attention one (weights + that -layer's own f16 KV) — measured on frank-medium. The gap can jump 2.9 GiB in a -single step; the best integer split can sit 0.2 GiB from balance while its -neighbours are 1+ GiB off. An affine fit reproduces the deployed point and -overstates gaps in between (`tests/frank_medium_unit.bats` encodes the fit as -a gate; treat a NEGATIVE fitted constant as a unit bug). - -Knobs that move weight and REQUIRE a re-measure: `--ubatch-size` (compute -buffers are **~2x heavier on the output/drafter card** — 512→1536 added -~3.3 GiB there vs ~1.7 GiB elsewhere), a drafter swap (weights + draft KV), -`--ctx-size`/`--parallel` (KV). Total free VRAM is invariant across splits — -only the distribution moves; use that to sanity-check a suspicious row. - -**Procedure (one split per load):** - -```bash -WORK=/tmp/split.env; cp scripts/serve-.env $WORK -export FRANK_MEDIUM_ENV_FILE=$WORK -for ts in 39,26 38,27 37,28 36,29 35,30; do - sed -i "s|^TENSOR_SPLIT=.*|TENSOR_SPLIT=$ts|" $WORK - nohup bash scripts/serve-.sh >/tmp/split.log 2>&1 & - # poll /health (or process exit), then: - rocm-smi --showmeminfo vram | grep 'VRAM Total Used' # sample ~6 s after health - pkill -x llama-server; sleep 6 -done -``` - -Load → sample → kill, NO requests sent (allocation only). Run under `nohup` -and poll from a separate call (a load is 30–90 s). Pick the smallest -inter-card gap that clears the per-card floor with room to spare; one `-ts` -step ≈ 1.5 GiB at ~6k batch / 262k ctx on 30 GiB cards — enough to bracket, -not to solve. Prefer balance: the busy card's headroom bounds the context -(`ctx_extra ≈ (floor − measured_free) / per-position cost on the busier -card`). A faulted split (`SDMA0 UTCL2 page fault` → `illegal memory access` → -`ggml_abort`) is OFF-LIMITS — re-measure, never re-tune; more headroom does -not prevent a boundary-copy fault. - -## 6. Drafter KV: sized from the TARGET, bounded by its window policy - -`common/speculative.cpp` builds the draft context with -`cparams.n_ctx = llama_n_ctx(ctx_tgt)` — `common/speculative.cpp:2663` on -`master`, re-read 2026-10-18 — **no flag caps it.** So: - -``` -draft KV = ctx_tgt × block_count × 2 (K,V) × kv_heads × key_length × dtype -``` - -…unless the drafter declares a sliding window. Measured at 262,144 target -ctx: DFlash2 (`sliding_window = 2048`) ≈ 40 MiB; a same-shape 5-layer drafter -with NO window = 20 KiB/position = **5.0 GiB** — a four-layer swing in the -balance point, and swapping them invalidated an already-balanced split. - -**Failure signature:** the engine loads the whole model, answers `/health`, -and dies on the FIRST draft decode with `ROCm error: out of memory` / -`cudaMalloc failed` inside `common_speculative_impl_draft_dflash::process` — -a sizing failure, not a broken drafter; under a self-healing unit it presents -as a crash-loop. Read `sliding_window`, `block_count`, `head_count_kv`, -`key_length` from the drafter's GGUF header first. A compact reader (the -three format traps: no KV alignment — v3 alignment applies only to the tensor -section; arrays are `type(9) → element_type(u32) → count(u64)`, element type -FIRST; the enum is `7=BOOL 9=ARRAY`, 64-bit = 10/11/12, and 8=STRING/9=ARRAY -is the v2/v3 file order, NOT gguf.h's 2/3): - -```python -import struct, sys -def rd(path): - f = open(path, 'rb'); assert f.read(4) == b'GGUF' - struct.unpack('