From 1ed7f0ddf243dd94aea2febbcdf1e91dc6de2f05 Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 19 Oct 2026 02:51:32 -0700 Subject: [PATCH] docs(agents): compress Key Lessons to one law per bullet, correct the FA-quants law MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Key Lessons Learned 483 -> 340 lines. Every law survives; only the essays went. Port-before-cut census over the diff: 454 deleted lines, 0 orphaned facts (every number still has a holder in the vault). - Each lesson is now one bullet with the page that owns the detail. - De-duplicated the page's own internal repeats: the layer-split law was told twice (PCIe and ROCm sections), the FA-quants law twice more. - CORRECTION, verified on dankclaw 2026-10-19: `GGML_CUDA_FA_ALL_QUANTS` is deprecated upstream (master's ggml/cmake/common.cmake tells you to use GGML_CUDA_FA_QUANTS=all), the successor's default pair list `q4_0-q4_0;q8_0-q8_0;f16-f16;bf16-bf16` compiles the quantized-KV kernels with the old flag OFF (master build-cuda cache: FA_ALL_QUANTS=OFF), and neither llama tree references the old macro under ggml/src/ggml-cuda/. The "required or prefill collapses ~12x" law is obsolete; the new law is to read the CMakeCache pair list. - frank-zealot: ported the three facts AGENTS.md alone carried, and verified them against the host first — the 2026-10-18 13:16 cutover boot really did load PTQ1_0 at `new slot, n_ctx = 133888` x3 (journal), the mmproj fit check really printed 873.10 MiB against a 21,439 MiB projection, and the warmup spread is 2.6 -> 1,258 t/s prefill. - New tooling: scripts/check-okf.py (OKF v0.1 conformance linter) and scripts/gen-vault-index.py (root index regenerated from frontmatter). Co-Authored-By: omp --- AGENTS.md | 765 +++++++++------------- scripts/check-okf.py | 175 +++++ scripts/gen-vault-index.py | 135 ++++ serverAppConfigs-docs/LLM/frank-zealot.md | 15 +- 4 files changed, 631 insertions(+), 459 deletions(-) create mode 100755 scripts/check-okf.py create mode 100755 scripts/gen-vault-index.py diff --git a/AGENTS.md b/AGENTS.md index e5c817b..b41f980 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -475,481 +475,338 @@ linked runbooks. **PCIe / multi-GPU** -- **RCCL collectives need PCIe atomics on every participating GPU.** RCCL's - kernels carry a hidden hostcall-buffer parameter; ROCr refuses to dispatch - them when the device has no atomics, and the failure surfaces as - `hipErrorIllegalState` ("the operation cannot be performed in the present - state") from `hipLaunchKernel`, not as anything that names atomics. The - value is consumed at driver probe time, so writing the AtomicOp Requester - bit with `setpci` at runtime changes nothing. Check the *chain*: - `lspci -vv -s ` and every upstream bridge must show - `AtomicOpsCap: Routing+` (a `Routing-` bridge is a hardware wall), and the - endpoint's Device Control 2 (`setpci -s CAP_EXP+28.w`) must have bit - 6 set. AMD desktop platforms put one card on CPU root ports (atomics work) - and the rest behind the chipset (usually not) — that asymmetry is what - blocked frank-medium's TP=2. A one-card collective proves nothing: NCCL - short-circuits at `world_size=1`. -- **That wall was physical, and it was fixed on 2026-09-20 by a board swap.** - Both paths that put two cards on CPU root ports were: an M.2→PCIe x4 riser - into a *CPU-attached* M.2 slot, or a board whose CPU lanes bifurcate to - x8/x8. dankserver took the second: **ASUS ProArt B850-Creator WiFi NEO**, - `PCIEX16_1` + `PCIEX16_2` at x8/x8 Gen5 on root ports `00:01.1` / `00:01.3`, - every bridge now `Routing+` and **both endpoints `ReqEn+`** — the exact - precondition the old test plan named. Bandwidth was never the constraint (a - 27B TP=2 allreduce at batch 4 is ≈10 MB per token-step, ≈1.4 ms at Gen4 x4 - and under 0.4 ms at Gen5 x8). - **Proven on the same day.** vLLM's R4D one-shot allreduce runs TP=2 here and - frank-medium serves on it; llama.cpp's RCCL path was built too - (`GGML_HIP_RCCL=ON`) and its tensor mode measured — real collective, slower - shape. Note the collective choice matters: TheRock's RCCL builds lack gfx1201 - device code (`invalid device function`), which is why R4D is the path vLLM - uses at all. Evidence matrix and the measured three-arm verdict: - [[Dual-R9700-Cooperation]]. -- **When a GPU collective dies, read the runtime, not the framework.** The - message that identified the cause came from `AMD_LOG_LEVEL=3` - (`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch - failed!`), not from vLLM or RCCL. `AMD_LOG_LEVEL` costs one env var and - prints the failing HIP call with its return code. -- **The software path that puts both cards on ONE model without a collective: - llama.cpp's layer split.** On the old board vLLM could not do it in any of its - parallelisms — TP needs an allreduce, PP needs the same p2p primitive, and both - died on the card that could not dispatch an RCCL kernel. `--split-mode - layer` in llama.cpp needs no - collective: layers are distributed in order and only the boundary activation - crosses PCIe, once per forward pass. Measured here: 1991 t/s prefill / - 18.8 tok/s decode layer-split, against 272 / 26.3 tensor (butterfly - fallback). - This was what `frank-medium` ran until 2026-09-20, when the board swap made - collectives legal and radiance TP=2 beat it on every axis (95.8 vs 45.7 t/s - decode, 0.21 vs 0.72 s TTFT, 259 vs 72 t/s burst-4) — see - [[frank-medium-radiance-mxfp4]]; the llama shape is the rollback target - [[frank-medium-llamacpp-dflash2]]. -- **Layer split is also the cross-vendor answer — and `-ot` is what cannot be - combined with it.** On Dankclaw (2026-09-16) a CUDA 3090 + Vulkan iGPU layer - split ran a real two-device graph: `-dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85` - held 14,994 MiB on the 3090 and gave 27.2 tok/s with 268.5 t/s prefill at 64k - depth (iGPU alone: 25.8 / 231.8). What does NOT work is per-tensor placement - across devices: `--override-tensor =` sets - `has_tensor_overrides()`, a REQUIRED term of the scheduler's - `pipeline_parallel` flag (llama-context.cpp:433 — the only place in the tree - that reads it), and that flag is what sets `n_copies` in - `ggml_backend_sched_new`; with `n_copies == 1` a pre-allocated weight on a - non-primary device has no backend and the graph aborts (`ggml-backend.cpp:941`, - reported with the leaf's own op name `(NONE)`). `=CPU` is the only override - target that works, because the CPU backend is last in the list. Pure Vulkan - changes nothing here — the gate is the override, not the driver mix. See - [[x4-pair-experiment]]. -- **A host↔3090 link at x4 measures 6.72 GB/s, and that is a design - constraint, not a detail.** `pcie-link-test.cu` on Dankclaw (2026-09-16, - services stopped): pinned H2D 6.72 GB/s, D2H 6.60 GB/s — 85 % of the Gen4 x4 - ceiling — with async no better than sync and pageable within 1 % of pinned, - i.e. the copies are link-limited rather than staging-limited (the usual - "pageable costs 2x" rule does not apply here). The card is x16-capable and the - Strix Halo port beneath it is x8-capable (`LnkCap: 16GT/s, Width x8` on - `00:03.1`), so the x4 lives in the adapter or cable; the link idles at - 2.5 GT/s x4 and trains up under load. Consequence for any engine that - *streams* weights across it: a 3 MiB slab costs 474 us, and a 111 GiB A3B - model needs ~1.70 GiB of active experts per token ⇒ **3.7 tok/s** (4.6 with a - 20 % resident cache), against **20.46 tok/s** for the same model with the - experts *computed* in host RAM by llama.cpp (`-ot exps=CPU`, whence the link - carries activations only) and 25–26.5 on the iGPU. Streaming over PCIe loses; - computing in RAM does not care. FreeToken — expert LRU in host RAM, exactly - this streaming shape — also loads only modelopt block-FP8 safetensors for - `qwen3_5_moe`, so a Q4_K_M GGUF is unloadable there before bandwidth is even - the question. See [[x4-pair-experiment]]. +- **RCCL collectives need PCIe atomics on every participating GPU, and the failure never names + atomics.** The kernels carry a hidden hostcall-buffer parameter; ROCr refuses to dispatch them on + a device without atomics and the error surfaces as `hipErrorIllegalState` from `hipLaunchKernel`, + not as anything that names atomics. The value is consumed at driver PROBE time, so writing the + AtomicOp Requester bit with `setpci` at runtime changes nothing. Check the CHAIN: `lspci -vv -s + ` and every upstream bridge must show `AtomicOpsCap: Routing+` (a `Routing-` bridge is a + hardware wall), and the endpoint's Device Control 2 (`setpci -s CAP_EXP+28.w`) must have bit + 6 set. AMD desktop platforms put one card on CPU root ports (atomics work) and the rest behind the + chipset (usually not) — that asymmetry is what blocked TP=2 here. A one-card collective proves + nothing: NCCL short-circuits at `world_size=1`. [[PCIe Link and Atomics]] +- **That wall was physical, and a board swap removed it on 2026-09-20.** dankserver took the path of + a board whose CPU lanes bifurcate to x8/x8 Gen5, both cards on root ports, every bridge `Routing+` + and BOTH endpoints `ReqEn+` — the exact precondition the old test plan named. Bandwidth was never + the constraint (a 27B TP=2 allreduce at batch 4 is ≈10 MB per token-step, under 0.4 ms at Gen5 x8). + The collective choice matters: TheRock's RCCL builds lack gfx1201 device code + (`invalid device function`), which is why vLLM runs its R4D one-shot allreduce here at all. + Evidence matrix and the measured three-arm verdict: [[Dual-R9700-Cooperation]]. +- **When a GPU collective dies, read the runtime, not the framework.** The message that identified + the cause came from `AMD_LOG_LEVEL=3` (`Pcie atomics not enabled, hostcall not supported` → + `AQL dispatch failed!`), never from vLLM or RCCL — the env var costs one line and prints the + failing HIP call with its return code. [[PCIe Link and Atomics]] +- **Layer split is the no-collective path, on one vendor's cards and across two — and + `--override-tensor` is what cannot ride with it.** Layers go in order and only the boundary + activation crosses PCIe once per forward pass, so it works wherever a collective cannot be + dispatched (and on an RCCL-less fork it beats tensor mode, which falls back to a butterfly + allreduce); it is also the cross-vendor answer, where a CUDA + Vulkan split held a real + two-device graph. But `--override-tensor =` sets `has_tensor_overrides()`, a + REQUIRED term of the scheduler's `pipeline_parallel` flag (llama-context.cpp), and that flag sets + `n_copies` in `ggml_backend_sched_new`: with `n_copies == 1` a pre-allocated weight on a + non-primary device has no backend and the graph aborts (ggml-backend.cpp, reported with the leaf's + own op name `(NONE)`). `=CPU` is the only override target that works, because the CPU backend is + last in the list, and pure Vulkan changes nothing — the gate is the override, not the driver mix. + Pin the spec-decode drafter to one device: an unpinned draft under tensor mode aborts the KV build, + and on a split its home is the card holding the target's last layers. + [[Llama Cpp Multi-Device Splits]] [[x4-pair-experiment]] +- **A host↔3090 link at x4 measures 6.72 GB/s, and that is a design constraint, not a detail.** + Pinned H2D 6.72 / D2H 6.60 GB/s (≈85 % of the Gen4 x4 ceiling), async no better than sync and + pageable within 1 % of pinned — the copies are LINK-limited, not staging-limited, so the usual + "pageable costs 2×" rule does not apply. The card and the port beneath it are both wider than x4, + so the x4 lives in the adapter or cable, and the link idles at 2.5 GT/s until load. Consequence + for any engine that STREAMS weights across it: a 3 MiB slab costs 474 µs, so a 111 GiB A3B model + needing ~1.70 GiB of active experts per token lands at 3.7 tok/s, against 20.46 tok/s for the same + model with the experts COMPUTED in host RAM (`-ot exps=CPU`, where the link carries activations + only). Streaming over PCIe loses; computing in RAM does not care. [[x4-pair-experiment]] **ROCm / GPU** -- **Layer split needs no collective** — layers go in order and only the boundary - activation crosses PCIe, so it is the fallback wherever a collective cannot be - dispatched (and on an RCCL-less fork it beats tensor mode, which falls back to - a butterfly allreduce). Pin the spec-decode drafter to one device: an unpinned - draft under tensor mode aborts the KV build, and on a split its home is the - card holding the target's last layers. Deployed history: - [[frank-medium-llamacpp-dflash2]]. -- **Bundling a ROCm runtime next to a llama.cpp binary: `cp -r SRC DST/` is a - trap.** With a missing `DST`, cp creates `DST` as a copy of `SRC` and drops - one directory level. rocBLAS and hipBLASLt look for their device code at a - fixed depth (`../lib/rocblas/library`, `../lib/hipblaslt/library/`), - and the flattened copy fails at *request* time, not load time: the engine - reads the weights, answers `/health` and dies on the first completion with - `Cannot read .../TensileLibrary.dat: No such file or directory for GPU arch - : gfx1201`. Declare the destination, then copy the contents. -- **mmap must stay on** for the big-PLE models: `--no-mmap` forces the - ~50 GiB PLE table into anonymous memory → silent cgroup kill, dmesg is the - only evidence. mmap pages ARE page cache. Never pin the PLE tensor with - `-ot` (23 → 13.4 t/s). -- **HIP reads bare env vars** `TEMP`, `HOST`, `GPU_MAX_HEAP_SIZE`. Serve - scripts export env files with `set -a`, so a tunable named `TEMP` leaks - into HIP (`TEMP=1.0` sent the JIT cache to path `1.0` and segfaulted every - load, fixed 2026-08-13 by renaming to `TEMPERATURE`). Never name a tunable - these; check with +- **Bundling a ROCm runtime next to a llama.cpp binary: `cp -r SRC DST/` is a trap.** With a missing + `DST`, cp creates `DST` as a copy of `SRC` and drops one directory level. rocBLAS and hipBLASLt + look for their device code at a FIXED depth (`../lib/rocblas/library`, + `../lib/hipblaslt/library/`), so the flattened copy fails at REQUEST time, not load time: + the engine reads the weights, answers `/health` and dies on the first completion with + `Cannot read .../TensileLibrary.dat: No such file or directory for GPU arch : gfx1201`. Declare + the destination, then copy the contents. +- **mmap must stay on** for the big-PLE models: `--no-mmap` forces the ~50 GiB PLE table into + anonymous memory → silent cgroup kill, with dmesg as the only evidence. mmap pages ARE page cache, + and pinning the PLE tensor with `-ot` cost 23 → 13.4 t/s. [[Llama Cpp Serving Sizing]] +- **HIP reads bare env vars** `TEMP`, `HOST`, `GPU_MAX_HEAP_SIZE`. Serve scripts export env files + with `set -a`, so a tunable named `TEMP` leaks into HIP (`TEMP=1.0` sent the JIT cache to path + `1.0` and segfaulted every load, fixed 2026-08-13 by renaming to `TEMPERATURE`). Never name a + tunable these; check with `strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE"`. -- **`-DGGML_CUDA_FA_ALL_QUANTS=ON` is required** for q8_0 KV + flash - attention on CUDA builds; without it prefill collapses ~12x (95 vs 1300+ - t/s, measured 2026-08-17). -- **Flash-Next MTP heads do not load on any current loader** — they need full - per-layer `blk..hc_attn_norm.weight` and ship only `blk.48` (verified - 3x). Don't re-attempt without new loader support. -- **QSA flash-attn assert:** `GGML_ASSERT(a-type == GGML_TYPE_F32)` on - qwen4exp needs `scripts/patches/qwen4exp-qsa-f32-mask.patch` — but only on - the agentionai FAST layer set; the kingjones STRIX set works unpatched. -- **GPU budget on Dankclaw is the GTT aperture (~114 GiB), not the - rocm-smi "dedicated" figure.** The Radeon 8060S (gfx1151) reports - 116,732 MiB (`llama-server --list-devices`; - `gtt_total=122402963456` in `/sys/class/drm/card1/device/`). The older - "64 GiB, SVM-backed" lesson was already stale — the CIRU Orca predecessor - held 92.6 GiB of weights on the iGPU. `free`/`dmidecode` show the - CPU-side carve-out — intentional, do not change. +- **`GGML_CUDA_FA_ALL_QUANTS` is dead; do not build for it.** Verified on this host 2026-10-19: + master's `ggml/cmake/common.cmake` warns it is deprecated in favour of `GGML_CUDA_FA_QUANTS`, the + successor's DEFAULT pair list (`q4_0-q4_0;q8_0-q8_0;f16-f16;bf16-bf16`) already compiles the + quantized-KV kernels with the old flag OFF, and neither llama tree on this host references the old + macro anywhere under `ggml/src/ggml-cuda/`. The old "~12× prefill collapse" belongs to the older + source line. Prove a build's KV ability from its `CMakeCache.txt`. [[frank-zealot]] +- **Flash-Next MTP heads do not load on any current loader** — they need full per-layer + `blk..hc_attn_norm.weight` and ship only `blk.48` (verified 3×). Do not re-attempt without new + loader support. +- **QSA flash-attn assert:** `GGML_ASSERT(a-type == GGML_TYPE_F32)` on qwen4exp needs + `scripts/patches/qwen4exp-qsa-f32-mask.patch` — but only on the agentionai FAST layer set; the + kingjones STRIX set works unpatched. [[Llama Cpp Serving Sizing]] +- **The GPU budget on Dankclaw is the GTT aperture, not the rocm-smi "dedicated" figure.** The + Radeon 8060S (gfx1151) reports 116,732 MiB (`llama-server --list-devices`; `gtt_total` in + `/sys/class/drm/card1/device/`), and the older "64 GiB, SVM-backed" lesson was already stale. + `free`/`dmidecode` show the CPU-side carve-out — intentional, do not change. [[Dankclaw System Specs]] **Load / crash diagnosis** -- Model load is slow (minutes cold); health returns 503 while loading; do - not kill the process. `deploy.sh` waits up to 20 minutes (`LLM_WAIT`). On - halogen, the FIRST request after a (re)start can go silent ~45–60 s - (health `engine_unresponsive`, 503s) and then self-recovers — do not - restart over it. -- Crash-loop procedure: (1) `journalctl -u -n 100` + kernel line via - `journalctl -k` (segfault at address 8 in `libamdhip64` = NULL unref in - HIP); (2) run serve script, then raw binary — different outcome isolates - the env file; (3) `gdb -batch -ex run -ex "bt 25" --args bash - scripts/serve-.sh` — llama-server buffers stdout and loses the log - tail on SIGSEGV; trust gdb, not the log; (4) `amdgpu: SVM mapping failed` - spam is a symptom, not a root cause. -- Journal signatures: client abort logs `W srv stop: cancel task, id_task = - N`; natural stop logs only `slot release` with `truncated = 0`. The server - never preempts a running generation; excess tasks queue when slots are - busy. +- Model load is slow (minutes cold) and health returns 503 while it runs: do NOT kill the process, + `deploy.sh` waits up to 20 minutes (`LLM_WAIT`). On halogen the FIRST request after a (re)start + can go silent for ~45–60 s (`engine_unresponsive`, 503s) and then self-recovers — do not restart + over it. [[frank-dragoon]] +- Crash-loop procedure: (1) `journalctl -u -n 100` AND the kernel line via `journalctl -k` (a + segfault at address 8 in `libamdhip64` is a NULL unref in HIP); (2) run the serve script, then the + raw binary — a different outcome isolates the env file; (3) `gdb -batch -ex run -ex "bt 25" + --args bash scripts/serve-.sh`, because llama-server buffers stdout and loses the log tail on + SIGSEGV: trust gdb, not the log; (4) `amdgpu: SVM mapping failed` spam is a symptom, not a root + cause. [[LLM Performance Diagnosis]] +- Journal signatures: a client abort logs `W srv stop: cancel task, id_task = N` while a natural stop + logs only `slot release` with `truncated = 0`. The server never preempts a running generation; + excess tasks queue when slots are busy. **Host memory / swap** -- Swap resize ordering: create and `swapon` the replacement BEFORE `swapoff` - of the old one, or live pages must land in RAM that `MemAvailable` - overstates. Dankclaw swap (64 GiB at `/swap.img`, fstab-driven) protects - the co-tenants, not frank-dragoon — pinned weights don't swap, and paging - the KV pool is worse than the OOM it prevents. -- Host swap state is invisible to `deploy.sh` (no repo file) — check - `swapon --show` before reasoning about headroom. -- **A sysfs raise is a countdown to the next reboot.** `ttm.pages_limit` defaults - to half of RAM, so the manual raise to 200 GiB (2026-10-05) vanished with the - 2026-10-12 23:17 POWER FAILURE on dankserver: the cap fell to 125,988 MiB, - frank-archon's pinned-ledger preflight (floor 150,000 MiB) refused the start, - and `Restart=always` re-ran that refusal **6,919 times in ~20 h** — a unit in - `activating (auto-restart)` that exits 1 in 50 ms with no container and no - engine log is a PREFLIGHT refusal, not an engine fault. Repo law now: - `scripts/ttm-preflight.sh` (sourced by `serve-frank-archon.sh`) raises the knob - to `FRANK_ARCHON_TTM_TARGET_MIB` itself, re-reads, and refuses by name only - after a write the kernel rejected. Never prescribe a manual sysfs write as the - recovery procedure for a service that can boot before a human reaches it. -- **Silent RAM corruption outranks every software theory.** frank-archon's - ~40k-token output corruption (2026-10-04/05, repetition/garbage in long - agent conversations) was marginal DDR5 at 4800 MT/s on four DIMMs, fixed - by 4600. Consumer non-ECC memory raises **no MCE** for a silent flip, and - a short in-band pattern check (60 s, 8 GiB of 246 GiB) is not a stability - proof — combined, those two instruments said "ruled out" while the RAM - was the cause. "4800 is the JEDEC floor on 5600-rated DIMMs" is a rating, - not a margin guarantee. When the fault is data-shaped (NaN states, - garbage tokens) and a software revert doesn't move it, price in the RAM - before the runtime — and remember a revert tested under the same marginal - hardware is itself confounded. See the runbook's 2026-10-05 section. +- **Swap resize ordering: create and `swapon` the replacement BEFORE `swapoff` of the old one**, or + live pages must land in RAM that `MemAvailable` overstates. Dankclaw's 64 GiB `/swap.img` + (fstab-driven) protects the co-tenants, not an endpoint with pinned weights — paging the KV pool + is worse than the OOM it prevents. Host swap state is invisible to `deploy.sh` (no repo file): + `swapon --show` before reasoning about headroom. [[Swap Configuration]] +- **A sysfs raise is a countdown to the next reboot.** `ttm.pages_limit` defaults to half of RAM, so + a manual raise vanished with a power failure, frank-archon's pinned-ledger preflight refused the + start, and `Restart=always` re-ran that refusal 6,919 times in ~20 h — **a unit in + `activating (auto-restart)` that exits 1 in 50 ms with no container and no engine log is a + PREFLIGHT refusal, not an engine fault.** Repo law now: `scripts/ttm-preflight.sh` raises the knob + to `FRANK_ARCHON_TTM_TARGET_MIB` itself, re-reads, and refuses by name only after a write the + kernel rejected. Never prescribe a manual sysfs write as the recovery procedure for a service that + can boot before a human reaches it. [[frank-archon]] +- **Silent RAM corruption outranks every software theory.** frank-archon's long-conversation + repetition/garbage was marginal DDR5 at 4800 MT/s on four DIMMs, fixed by dropping to 4600. + Consumer non-ECC memory raises NO MCE for a silent flip, and a short in-band pattern check + (60 s over 8 GiB of 246 GiB) is not a stability proof — combined, those two instruments said + "ruled out" while the RAM was the cause, and "4800 is the JEDEC floor" is a rating, not a margin + guarantee. When the fault is data-shaped (NaN states, garbage tokens) and a software revert does + not move it, price in the RAM before the runtime — and remember a revert tested under the same + marginal hardware is itself confounded. [[log]] **Runtime-specific guards** -- **Rootless podman cannot RAISE rlimits, and `--ulimit nofile=1048576` - dies at container create, not at runtime.** runc reports - `error setting rlimit type 7: operation not permitted` (type 7 = - RLIMIT_NOFILE) when the requested hard limit exceeds the host process's - hard limit — on Dankclaw that ceiling is 524288. Inherited-equal limits - are fine, so `memlock=-1:-1` works iff the unit grants - `LimitMEMLOCK=infinity` (systemd is privileged; the value flows through - podman to runc without any raise). Upstream halogen's own run lines pass - NO `--ulimit` at all — copy vendor run lines before adding flags - (measured 2026-10-05 on the frank-dragoon cutover). -- **Kairic Edge** (if ever re-deployed): the pinned CK commit needs - `third_party/composable_kernel/include/ck/config.h` generated manually - (untracked; `git clean` removes it), and its greedy-argmax fastpath rejects - non-greedy requests and reasoning budgets. Re-test chat before production. -- **FastMTP sidecar**: build must be commit `4df29be4` + the HauhauCS patch; - stock binaries fail the draft load with a tensor-shape mismatch. -- **A GGUF's custom tensor type picks the runtime — read the histogram before - promising a file to a loader.** `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is - ftype 115 with 311 tensors of type 103 (`GGML_TYPE_Q8_0_ROCMFPX`); stock - llama.cpp and gufo refuse it at load, so ROCmFPX was not a preference but - the only door. Counting tensor types out of the GGUF KV section takes one - python read and settles a runtime argument in minutes. The build identity - matters as much as the fork: the 2026-08-25 ROCmFPX builds SIGSEGV in - `ggml_compute_forward_set_rows` on this same file, so - `scripts/build-rocmfpx.sh` pins commit `36e9acd` by SHA. -- **Speculative-decode flags move as a SET.** Handing the ROCmFPX fork - `--model-draft` with no `--spec-type` (the naive way to build a "no MTP" - bench arm) segfaulted at load, core dumped, no diagnostic (2026-10-02). A - baseline arm must strip `--model-draft` AND every `--spec-draft-*` AND - `--spec-type`, or it measures a crash instead of the absence of drafting. -- **An orphaned scratch server taxes the whole host.** A bench child left - alive by a client-side exception kept ~26 GiB of weights resident on a - 121 GiB box, pushed `available` to 3 GiB, and every later rate was suspect. - Record the child pid and kill it in a `finally`, then prove `free` came back - before believing a number. Kill by pid — never `pkill -f llama-server`, - which also hits frank-zealot. +- **Rootless podman cannot RAISE rlimits, so `--ulimit nofile=1048576` dies at container CREATE**, + not at runtime: runc reports `error setting rlimit type 7: operation not permitted` (type 7 = + RLIMIT_NOFILE) when the requested hard limit exceeds the host process's ceiling (524288 on + Dankclaw). Inherited-equal limits are fine, so `memlock=-1:-1` works iff the unit grants + `LimitMEMLOCK=infinity` — systemd is privileged and the value flows through podman to runc without + any raise. Upstream halogen's own run lines pass NO `--ulimit` at all: copy vendor run lines + before adding flags. [[frank-dragoon]] +- **A GGUF's custom tensor type picks the runtime — read the histogram before promising a file to a + loader.** `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is ftype 115 over 311 tensors of type 103 + (`GGML_TYPE_Q8_0_ROCMFPX`), so stock llama.cpp and gufo refuse it at load and the fork was not a + preference but the only door. Counting tensor types out of the GGUF KV section takes one python + read and settles a runtime argument in minutes. Build identity matters as much as the fork: the + 2026-08-25 ROCmFPX builds SIGSEGV in `ggml_compute_forward_set_rows` on this same file, so + `scripts/build-rocmfpx.sh` pins commit `36e9acd` by SHA. [[GGUF Compatibility]] +- **Speculative-decode flags move as a SET.** Handing the fork `--model-draft` with no `--spec-type` + (the naive way to build a "no MTP" bench arm) segfaulted at load, core dumped, no diagnostic. A + baseline arm must strip `--model-draft` AND every `--spec-draft-*` AND `--spec-type`, or it + measures a crash instead of the absence of drafting. [[frank-zealot]] +- **An orphaned scratch server taxes the whole host.** A bench child left alive by a client-side + exception kept ~26 GiB of weights resident on a 121 GiB box and pushed `available` to 3 GiB, so + every later rate was suspect. Record the child pid, kill it in a `finally`, then prove `free` came + back before believing a number — and kill by pid, never `pkill -f llama-server`. + [[LLM Bench Windows and Measurement]] +- Retired-runtime traps that can come back: **Kairic Edge** needs + `third_party/composable_kernel/include/ck/config.h` generated manually (untracked, so `git clean` + removes it) and its greedy-argmax fastpath rejects non-greedy requests and reasoning budgets, so + re-test chat before production; the **FastMTP sidecar** must be commit `4df29be4` + the HauhauCS + patch, since stock binaries fail the draft load with a tensor-shape mismatch. **llama.cpp serving** -- **MTP is a property of the FILE, not of the flags.** The `-mtp` GGUF build - carries the head as `blk..nextn.*` (`qwen35.nextn_predict_layers = 1`); - `--spec-type draft-mtp` then sets `load_mtp`, and the draft context runs on - the MAIN model's weights (`shares_model`), so there is no second copy of a - 12 GiB model in VRAM and no `-md` argument. The plain build has no MTP - tensors at all, and pointing `--model-draft` at the file would load it twice. -- **What can be chained, what cannot, and what a chain costs on a hybrid.** - `--spec-type a,b` builds every enabled implementation and chains drafting - through them, in a fixed internal order (ngram types first, then the - drafters — CLI order is irrelevant). The gate is `ctx_dft`: **draftless** - types (the ngram family) are added unconditionally, while `draft-mtp`, - `draft-dflash`, `draft-dspark` and `draft-eagle3` all require a draft - CONTEXT. So a chain may hold one context-backed drafter plus any number of - ngram types — `draft-dflash,ngram-mod` is legal and measured (2.6x the - drafter alone on prompt-echo traffic, byte-identical to it on prose at the - fleet's 48/64 window) — but `draft-mtp,draft-dflash` dies at load: *"context - type MTP requested but model doesn't contain MTP layers"*, because MTP - builds its context against whatever model the draft context holds. Two - context-backed types cannot share one process, and MTP's head only loads - when MTP **is** the draft model. **A LEGAL chain can still lose on a - recurrent model:** `draft-mtp,ngram-mod` on frank-dragoon's qwen35 hybrid - measured −15 % prose / −7 % block-copy (2026-10-03) — adding the ngram layer - logs `recurrent state rollback is not compatible with 'ngram-mod' - - disabling rollback support`, and MTP verification depends on that rollback, - so acceptance drops on ALL traffic even where the matcher fires. Measure - chains per architecture, never by analogy. -- **A drafter's KV scales with the TARGET's context, and its window policy - decides the price.** The draft context is built at `llama_n_ctx(ctx_tgt)` with - no flag to cap it, so a drafter declaring NO sliding window caches the whole - window on `--spec-draft-device` and can OOM the first draft graph capture - AFTER `/health` has answered. Read `sliding_window` out of the drafter's GGUF - header before sizing, and prefer a skewed `-ts` over a context cut. Worked - example: [[frank-medium-llamacpp-dflash2]]. -- **A draft context bills its own f16 KV over the WHOLE POOL, and on a - single-card endpoint that squeeze can cost more than the drafter wins.** With - `--parallel N` the drafter pays the pool, not the slot, plus its own compute - buffers. So the MTP question is never "does the file carry the head" but "what - do draft KV + buffers cost against what the batch knobs leave for compute" — - a per-shape answer. Measure the arm, never the flag; the booted arm table is - on [[frank-zealot]]. -- **A quant's scale overhead belongs in its byte rate; doubling a measured - figure to "get the f16 number" invents 6.25 %.** q8_0 is 1.0625 B/element - (32 codes + one 2-byte scale per 32-block), f16 is exactly 2.0 ⇒ the ratio is - **1.882×**, not 2×. Derive byte rates from element count × block layout, then - check them against the boot ledger — this repo carried an invented 2× number - for weeks. Current KV math: [[frank-zealot]]. -- **`--ubatch-size` has an OPTIMUM, not a plateau, and the curve is - BACKEND-SPECIFIC — never carry it across hosts.** Its compute buffers land - about TWICE as heavy on the card that holds the output layers and the drafter, - so a split balanced at one batch is unbalanced at another: re-measure the - balance after any batch change. The logical `--batch-size` is nearly free. - A large-batch allocation can fail on BUFFER CONTIGUITY while totals still fit - — no totals model can see that. Ladders: [[frank-medium-llamacpp-dflash2]] - (R9700 split) and [[frank-zealot]] (CUDA 3090). -- **On the CUDA 3090 that same knob has no interior optimum up to 2048.** A - prompt shorter than the ubatch cannot see the knob at all — that is the - control that proves the knob, not the traffic, moved the number. When a bench - IS ordered, A/B it by restarting the SERVICE through a transient - `FRANK_ZEALOT_ENV_FILE` drop-in, never a scratch port: one card holds one - server. Numbers: [[frank-zealot]]. -- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same - size** (a DeltaNet boundary and a full-attention boundary can differ by ~1 GiB - each with its own KV), so balance by measurement. An affine fit reproduces the - deployed point and overstates the gaps in between. - [[frank-medium-llamacpp-dflash2]]. -- **ngram-mod prints no startup banner.** The load log shows the drafter's - `adding speculative implementation 'draft-dflash'` and nothing at all for the - matcher, so a chain can look unengaged while it is working. Identify it - behaviourally (`draft_n` > 0 only where text repeats) or by the per-task - `draft acceptance = ... mean len = ...` line. It costs no VRAM (a 64 MiB - host-RAM hash table) and its window decides which traffic it can see: too - short and it proposes on ordinary prose fragments, which costs verification - without acceptance. -- **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the - per-slot number is the contract — not `ctx-size`.** `n_ctx_slot = - CTX_SIZE / PARALLEL`, and llama.cpp refuses a prompt longer than the SLOT with - a 400 even when the pool has room, so `models.yml` `contextWindow` must follow - every reslice. Both slot directions cost or buy window, never pool; shorter - slots decode slightly faster (less attention to scan, one fewer recurrent - state). Measured reslices: [[frank-zealot]]. -- **Price a VRAM budget from two boots, never from a formula — and an mmproj - costs ~1.5× its file size** (the encoder's activation and compute buffers ride - with it). Subtracting two booted rows isolates it exactly; extrapolating - weights+KV+"a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7. - Then solve `KV ≤ card − non-KV − headroom` and round to a multiple of - `PARALLEL × 4096`. The two-boot table: [[frank-zealot]]. -- **`images` and `image_url` are NOT interchangeable: only the content part - reaches the model.** `--mmproj` loads, `usage.prompt_tokens` rises, and the - model still invents an answer if the client used the legacy top-level - `images` field on `/v1/chat/completions` — frank-zealot answered "Brown." for - a solid-red PNG that way and "Red." through the OpenAI `image_url` part. The - failure is silent because a hallucinated color is a plausible answer. Probe - vision with a solid color and expect an exact word. -- **`--jinja` is what `tools` needs.** On 2026-09-16 a subagent died with - `500 tools param requires --jinja flag`, from the ik_llama.cpp runtime that - then served frank-dragoon: every tool-using agent on that endpoint was broken - and the failure read like a model fault. Mainline llama.cpp 0.4.1 defaults - `--jinja` on and the serve script pins it; halogen also accepts `tools` and - returns `tool_calls`. Check tool support before routing a role at an endpoint. -- **Quantized KV on CUDA needs `GGML_CUDA_FA_ALL_QUANTS=ON`** — the same - option and the same ~12x prefill collapse described in the ROCm note above. - f16 KV needs nothing; frank-zealot ran f16 for that reason until - 2026-09-17, when `build-cuda` was rebuilt with the option ON so the - endpoint could take q8_0 KV. -- **llama-bench separates a LIST with `/` and SWEEP values with `,`.** Both - `-dev` and `-ts` follow this: `-dev CUDA0,Vulkan1` is two *single-device* - configurations, never one two-device run, and `-ts 0.85,0.15` is two sweep - values each becoming `[0.85, 0, …]` / `[0.15, 0, …]`, i.e. device 0 takes - everything. Write `-dev A/B -ts x/y`. This artifact produced a wrong fleet - conclusion on 2026-09-16 — "CUDA + Vulkan cannot serve one model", with two - bogus OOMs that were really "all 111 GiB on the 24 GiB card" — and the bench - table's own `dev` column was printing the truth the whole time. Any - placement-dependent number needs an independent check: sample the other +- **MTP is a property of the FILE, not of the flags.** The `-mtp` export carries the head as + `blk..nextn.*` (`qwen35.nextn_predict_layers = 1`); `--spec-type draft-mtp` sets + `load_mtp` and the draft context runs on the MAIN model's weights (`shares_model`), so there + is no second copy of a 12 GiB model in VRAM and no `-md` argument. The plain build has no MTP + tensors at all, and pointing `--model-draft` at that file loads it twice. [[frank-zealot]] +- **What can be chained: ONE context-backed drafter plus any number of draftless ngram types.** + `--spec-type a,b` chains in a fixed internal order (CLI order irrelevant) and the gate is + `ctx_dft`: the ngram family is added unconditionally while `draft-mtp`, `draft-dflash`, + `draft-dspark` and `draft-eagle3` all need a draft CONTEXT. So `draft-dflash,ngram-mod` is + legal (measured 2.6x the drafter alone on prompt-echo traffic) but `draft-mtp,draft-dflash` + dies at load — *"context type MTP requested but model doesn't contain MTP layers"*. **A LEGAL + chain can still lose on a recurrent model:** `draft-mtp,ngram-mod` on frank-dragoon's qwen35 + hybrid measured −15 % prose / −7 % block-copy, because adding the ngram layer logs + `recurrent state rollback is not compatible with 'ngram-mod' - disabling rollback support` and + MTP verification depends on that rollback. Measure chains per architecture, never by analogy. + [[frank-dragoon]] +- **A drafter's KV scales with the TARGET's context, and its window policy decides the price.** + The draft context is built at `llama_n_ctx(ctx_tgt)` with no flag to cap it, so a drafter + declaring NO sliding window caches the whole window on `--spec-draft-device` and can OOM the + first draft graph capture AFTER `/health` has answered. Read `sliding_window` out of the + drafter's GGUF header before sizing, and prefer a skewed `-ts` over a context cut. + [[Llama Cpp Multi-Device Splits]] +- **A draft context bills its own f16 KV over the WHOLE POOL, not the slot**, plus its own + compute buffers, so on a single-card endpoint that squeeze can cost more than the drafter + wins. The MTP question is never "does the file carry the head" but what draft KV + buffers + cost against what the batch knobs leave for compute. Measure the arm, never the flag. + [[frank-zealot]] +- **A quant's scale overhead belongs in its byte rate; doubling a measured figure to "get the + f16 number" invents 6.25 %.** q8_0 is 1.0625 B/element (32 codes + one 2-byte scale per + 32-block), f16 is exactly 2.0 ⇒ the ratio is **1.882×**, not 2×. Derive byte rates from element + count × block layout, then check them against the boot ledger — this repo carried an invented + 2× number for weeks. Current KV math: [[frank-zealot]] +- **`--ubatch-size` has an OPTIMUM, not a plateau, and the curve is BACKEND-SPECIFIC — never + carry it across hosts.** Its compute buffers land about TWICE as heavy on the card that holds + the output layers and the drafter, so a split balanced at one batch is unbalanced at another: + re-measure the balance after any batch change. The logical `--batch-size` is nearly free; a + large-batch allocation can fail on BUFFER CONTIGUITY while totals still fit, which no totals + model can see. A prompt shorter than the ubatch cannot see the knob at all — that is the + control that proves the knob, not the traffic, moved the number. The 2× R9700 split peaks at + 512; the CUDA 3090 has no interior optimum up to 2048. When a bench IS ordered, A/B it by + restarting the SERVICE through a transient `FRANK__ENV_FILE` drop-in, never a scratch + port: one card holds one server. Ladders: [[frank-medium-llamacpp-dflash2]] and + [[frank-zealot]]. +- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same size** (a DeltaNet + boundary and a full-attention boundary can differ by ~1 GiB each with its own KV), so balance + by measurement. An affine fit reproduces the deployed point and overstates the gaps in between. + [[Llama Cpp Multi-Device Splits]] +- **`ngram-mod` prints no startup banner.** The load log names `draft-dflash` and nothing for the + matcher, so a chain can look unengaged while it works. Identify it behaviourally (`draft_n` > 0 + only where text repeats) or by the per-task `draft acceptance = ... mean len = ...` line. It + costs no VRAM (a 64 MiB host-RAM hash table) and its window decides which traffic it can see: + too short and it proposes on ordinary prose fragments, costing verification without acceptance. + [[Llama Cpp Multi-Device Splits]] +- **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the per-slot number is + the contract — not `ctx-size`.** `n_ctx_slot = CTX_SIZE / PARALLEL`, and llama.cpp refuses a + prompt longer than the SLOT with a 400 even when the pool has room, so `models.yml` + `contextWindow` MUST follow every reslice. Both slot directions cost or buy window, never + pool; shorter slots decode slightly faster. Measured reslices: [[frank-zealot]]. +- **Price a VRAM budget from two boots, never from a formula — and an mmproj costs ~1.5× its file + size** (the encoder's activation and compute buffers ride with it). Subtracting two booted rows + isolates it exactly; extrapolating weights+KV+"a bit for buffers" is what leaves 1.1 GiB free + when you meant 2.7. Then solve `KV ≤ card − non-KV − headroom` and round to a multiple of + `PARALLEL × 4096`. **The projector is NOT billed in llama.cpp's own fit check**: the engine + prints its worst case and fits WITHOUT it, so price a shape from settled `nvidia-smi` after a + real boot — the projector, and any drafter context, lands after `/health` would already have + answered (same class as the DSpark drafter OOM). Two-boot table: [[frank-zealot]]. +- **`images` and `image_url` are NOT interchangeable: only the content part reaches the model.** + `--mmproj` loads, `usage.prompt_tokens` rises, and the model still invents an answer if the + client used the legacy top-level `images` field on `/v1/chat/completions` — frank-zealot + answered "Brown." for a solid-red PNG that way and "Red." through the OpenAI `image_url` part. + The failure is silent because a hallucinated color is a plausible answer; probe with a solid + color and expect an exact word. [[frank-zealot]] +- **`--jinja` is what `tools` needs.** A subagent died with `500 tools param requires --jinja + flag` on the ik_llama.cpp runtime that served frank-dragoon: every tool-using agent on that + endpoint was broken and the failure read like a model fault. Mainline llama.cpp defaults + `--jinja` on and the serve scripts pin it; halogen also accepts `tools` and returns + `tool_calls`. Check tool support before routing a role at an endpoint. +- **A model's own template can REJECT the harness's thinking level.** Qwen3.8's template accepts + only `xhigh`, `medium` and `low` for `reasoning_effort` and raises on anything else — HTTP 500, + not a clamp — so the harness's levels must be folded onto that set in `models.yml` + (`compat.reasoningEffortMap`). A role whose level is unmapped fails on its FIRST request and + the failure names Jinja, not the model: every subagent on `frank-zealot` died with + `Unexpected reasoning effort high` until `high: xhigh` was added. Halogen accepts the wider + set, so the two entries legitimately differ. +- **Thinking tokens count against `max_tokens`, so a short-budget vision probe reads as + blindness.** A probe at `max_tokens: 64` returned `content: ""` with `finish_reason: "length"` + and half a thought in `reasoning_content` while the image HAD reached the model (322 prompt + tokens vs 42 for the text alone). Budget ≥512 for any thinking model you probe for vision. +- **The FIRST request after a boot is a warmup artifact**, and the 3× spread on one shape makes + graph-capture cost read as a model fault. Benchmark the second request or later. + [[frank-zealot]] +- **A fork build hides its own dtype laws.** Vendor builds carry their own compile options, so + prove a build's KV behaviour from the KV buffer line in the boot ledger and its + `CMakeCache.txt` — never from the build flags you remember passing. [[frank-zealot]] +- **A llama.cpp endpoint is invisible to Prometheus unless `--metrics` is passed**: without it + the engine answers **501 Not Implemented** on `/metrics`, so the scrape fails while `/health` + answers 200 and `LlmEndpointDown` fires beside a healthy service — that is how `frank-zealot` + sat red across four payload generations. halogen serves `/metrics` natively, so the asymmetry + reads as "only the 3090 is broken". Diagnose with `lastError` from `/api/v1/targets`, not the + alert list, and note Alertmanager has no persistence: a reboot re-stamps `startsAt` on an alert + that has fired for weeks. [[Observability]] +- **A leftover bench drop-in silently hijacks the next deploy, and no health gate sees it.** The + sanctioned bench-window method (a transient `Environment=FRANK__ENV_FILE=/tmp/.env` + drop-in sourcing the repo env) outlives the bench unless retired: a leftover + `frank-zealot.service.d/ptq1.conf` made the cutover deploy boot a **different model at a + different shape** while `/health` answered 200, `/v1/models` echoed the alias (a flag, not a + fact) and the Playwright gate passed 33/33. After deploying any recently-benched LLM service: + `systemctl show -p DropInPaths` (must be empty), then + `tr '\0' ' ' -/` + `daemon-reload`. [[frank-zealot]] +- **A bench verdict that never reaches the repo is indistinguishable from a rogue arm.** That + same arm served production for 24 minutes with real numbers (~48 t/s decode, 801 t/s prefill at + 141k) that lived only in `/tmp` env files and journald, so the next agent read the surviving + drop-in as an unfinished experiment and reported a hijack. Write the measurement into the + endpoint page and the arm script's header in the session that takes it — `/tmp` and the journal + are not records. +- **`llama-bench` separates a LIST with `/` and SWEEP values with `,`** — for both `-dev` and + `-ts`. `-dev CUDA0,Vulkan1` is two *single-device* configurations, never one two-device run, + and `-ts 0.85,0.15` is two sweep values each putting everything on device 0. Write + `-dev A/B -ts x/y`. This artifact produced a wrong fleet conclusion ("CUDA + Vulkan cannot serve + one model") from two bogus OOMs, and the bench table's own `dev` column was printing the truth + the whole time. Any placement-dependent number needs an independent check: sample the other device's memory (`nvidia-smi`) while the run is loaded. -- **A model's own template can REJECT the harness's thinking level.** Qwen3.8's - template accepts only `xhigh`, `medium` and `low` for `reasoning_effort` and - raises on anything else — HTTP 500, not a clamp — so the harness's levels - must be folded onto that set in `models.yml` (`compat.reasoningEffortMap`). - A role whose level is unmapped fails on its FIRST request, and the failure - names Jinja, not the model: every subagent on `frank-zealot` died with - `Unexpected reasoning effort high` until `high: xhigh` was added on - 2026-09-16. Halogen accepts the wider set, so the two entries legitimately - differ. -- **The vision projector is NOT billed in llama.cpp's own fit check.** The - engine prints `[mtmd] estimated worst-case memory usage of mmproj is - 873.10 MiB` and then fits WITHOUT it: frank-zealot's Bonsai-2 boot projected - 21,439 MiB and settled at 22,664 (2026-10-17). Price a 3090-class shape from - settled `nvidia-smi` after a real boot, never from - `model + context + compute` — the projector (and any drafter context) lands - after `/health` would already have answered. Same failure class as the - DSpark drafter OOM. -- **The FIRST request after a boot is a warmup artifact.** Measured on - Bonsai-2/3090: 2.6 t/s prefill and 37.7 t/s decode on request #1, then - 1,258 t/s prefill and 54.7 t/s (per stream, two concurrent) on the next — a - 3× spread on the same shape. Benchmark the second request or later, or the - graph-capture cost reads as a model fault. -- **Thinking tokens count against `max_tokens`, so a short-budget vision probe - reads as blindness.** The Bonsai-2/`image_url` probe at `max_tokens: 64` - returned `content: ""` with `finish_reason: "length"` and half a thought in - `reasoning_content` — the image HAD reached the model (322 prompt tokens vs - 42 for the text alone). At 512 tokens the same image answers "Red". Budget - ≥512 for any thinking model you probe for vision. -- **Quantized KV is not the only dtype law a fork build can hide.** Vendor - builds carry their own compile options: frank-zealot's PrismML prebuilt - served `--cache-type-v q8_0` under flash attention at the exact 34,816 - B/token (the `GGML_CUDA_FA_ALL_QUANTS` door), so a build decision must be - proven by the KV buffer line in the boot ledger, not by the build flags you - remember passing. -- **A llama.cpp endpoint is invisible to Prometheus unless `--metrics` is passed.** - Without it the engine answers **501 Not Implemented** on `/metrics`, so the scrape - fails while `/health` answers 200 and `LlmEndpointDown` fires beside a healthy - service — that is how `frank-zealot` sat red across four payload generations - (the oldest retained sample of `up{job="frank-zealot"}`, 2026-10-11, is already - 0). halogen serves `/metrics` natively, so the asymmetry reads as "only the 3090 - is broken". Diagnose with `lastError` from `/api/v1/targets`, not the alert list, - and note Alertmanager has no persistence: a reboot re-stamps `startsAt` on an - alert that has been firing for weeks. -- **A leftover bench drop-in silently hijacks the next deploy, and no health - gate sees it.** The sanctioned bench-window method — a transient - `Environment=FRANK__ENV_FILE=/tmp/.env` drop-in, the arm env - `source`ing the repo env and then overriding it — outlives the bench unless - it is retired. On 2026-10-18 a leftover `frank-zealot.service.d/ptq1.conf` — - the arm Frank had ordered at 02:36 and measured from 03:14 — made the cutover - deploy boot a **different model at a different shape** (3 slots of 133,888, - not 3 × 262,144) while `/health` answered 200, `/v1/models` echoed the alias - (a flag, not a fact) and the Playwright gate passed 33/33. After deploying any - recently-benched LLM service: `systemctl show -p DropInPaths` (must be - empty), then `tr '\0' ' ' -/` + - `daemon-reload`. -- **A bench verdict that never reaches the repo is indistinguishable from a - rogue arm.** Same window: PTQ1_0 served production for 24 minutes with real - numbers (~48 t/s decode, 801 t/s prefill at 141k), but the figures lived only - in `/tmp` env files and journald, so the next agent on the host read the - surviving drop-in as an unfinished experiment and reported a hijack. Write the - measurement into the runbook and the arm script's header in the session that - takes it — `/tmp` and the journal are not records. **Ops** -- Never `pkill -f` a pattern that appears in your own command line — it kills - your ssh session. Use `pkill -x `. -- Both hosts allow passwordless sudo for the deploy user. Use plain `sudo` in - committed scripts (no `echo pw | sudo -S`); `FRANK_SUDO_PW` covers hosts - without it. -- ComfyUI on Dankclaw runs with **GPU access** (`/dev/kfd` + `/dev/dri` via - `AddDevice`, `GroupAdd=keep-groups`, plus the udev rule that makes those nodes - world-writable — user-unit spawns carry no supplementary groups). It must NOT - pass `--cpu`: the gfx1151 ROCm wheels exist and the unit's `--cpu` era ended - 2026-09-16, when a stale device-less container was found restarted 9,146 - times (`RuntimeError: No HIP GPUs are available`). -- SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its - ports off the gluetun service. -- Dankclaw boots **`iommu=pt`** since 2026-10-18. The boot line is tracked in - `dankclaw/host/grub` and moves ONLY through `scripts/set-dankclaw-bootline.sh` - (diff, install, `update-grub`, verify `grub.cfg`, `--reboot`) — never by - hand-editing `/etc/default/grub`. It replaced `amd_iommu=off` (2026-09-11), - adopted on the vendor's 13–16 % prefill figure and never measured here: the - local A/B on byte-identical traffic says identity mode costs **nothing** - (+2.0 %, 1,616 t/s cold prefill), so the XDNA2 NPU returned for free — - `amdxdna` binds `c6:00.1` (PASID width 10, identity group 26) and the old - `err=-19` only ever asked for an enabled IOMMU. **AMD PerfOpt** (iommu tree - 2026-09-24, Linux 7.4) still skips the page walk outright for the privileged - iGPU, but it arms only in the identity domain and 26.04 caps at `7.0.0-38`. -- Rootless Quadlet containers run with `--replace --rm`: `podman logs` covers - only the CURRENT instance; history across recreations lives in - `journalctl --user -u .service`. Read rootless storage WITHOUT sudo - (per-user storage — `sudo podman logs` finds nothing). -- **A dead rootless podman pause process breaks ALL container management on the - host, and `podman system migrate` panics in 5.7.0.** Symptoms: every `podman` - command fails with `invalid internal status … could not find any running - process`, every container start fails binding podman's bridge addresses - (`Error starting server failed to bind udp listener on 10.89.x.1:53: Cannot - assign requested address`), and every `Restart=always` quadlet loops. Container - storage is intact; what is gone is `/run/user//libpod/tmp/alive` (a UUID - token, not a pid) and the rootless netns holding the bridges — deleting the - stale token is not enough, and `podman system migrate` segfaults in - `teardownNetworkBackend`. Repair: **reboot** (`/run/user` clears and all 28 - quadlets return on their own — check `Linger=yes` and `WantedBy=default.target` - first). Seen 2026-09-16, hours into heavy GPU work on the same host — but the - trigger is NOT established (`journalctl -k` recorded no OOM kill; the kernel - had no other word on it). Treat the pause process as host state that can die - for its own reasons, and reach for the reboot when it does. -- **A container cannot reach a `Network=host` service at the host's LAN IP.** - netavark DNATs *published* ports only: from a bridge container - `192.168.0.143:` answers while `192.168.0.143:` and the - gateway `10.89.1.1` are both refused. Use `host.containers.internal` - (169.254.1.2). This is what 502'd `hermes.franksdank.cloud` until the Caddyfile - upstream was changed on 2026-09-16 (its vhost comment had claimed the LAN IP - was open from that bridge — it never was). -- **A `*.gguf` filter misses mradermacher's payloads: large quants ship as - `.partNofM` fragments.** `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF` - carries Q4_K_M (37.00 + 37.00 + 36.97 = 110.97 GiB) and Q4_K_S (35.00 + 35.00 - + 34.09 = 104.09 GiB) as parts, while the smaller rungs (Q2_K, Q3_K_*, IQ3_*, - IQ4_XS) are single files. Fetch the parts, verify each against the Hub's LFS - sha256, concatenate in numeric order — that is how the deployed 111 GiB file - was assembled. Listing the Hub tree and filtering `endswith(".gguf")` makes - Q4_K_S look like it does not exist. -- **A fetch script must `-f` and verify, or it reports success on an access - error.** `curl -o FILE URL` exits 0 on HTTP 401/403 and writes the error body - as the "artifact": a gated repo yielded three 153-byte shards with identical - sha256 values and a cheerful `DOWNLOAD_DONE` in 20 seconds for 104 GiB. Use - `-f`/`--fail-with-body`, compare each file's byte count to the Hub's declared - size, and check the LFS sha256 — `scripts/download-frank-medium-gguf.sh` - already does all three; ad-hoc fetches tend not to. Gated repos (e.g. - `orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF`, same 104.09 GiB Q4_K_S) - return 403 even with the host's HF token until that account accepts the gate. -- **A scout verdict is a lead, never a finding.** In the 2026-10-19 vault de-dup - campaign, four read-only scouts worked the same pages: one classified era pages as - stale *because* they document retired payloads (that is an era page's content), one - reported files that do not exist, one reported duplications that grep disproved - (`457.9 t/s` and the ping-RTT table are solo-held by `Ethernet Bandwidth Math`), - and one cited a line number past the end of the file it judged. Re-verify every - claimed duplicate and every claimed staleness against the file and today's fleet - state before editing; a scout yield buys no evidence. +- Never `pkill -f` a pattern that appears in your own command line — it kills your ssh session. + Use `pkill -x `. +- Both hosts allow passwordless sudo for the deploy user. Use plain `sudo` in committed scripts + (no `echo pw | sudo -S`); `FRANK_SUDO_PW` covers hosts without it. +- **ComfyUI on Dankclaw runs with GPU access** (`/dev/kfd` + `/dev/dri` via `AddDevice`, + `GroupAdd=keep-groups`, plus the udev rule that makes those nodes world-writable — user-unit + spawns carry no supplementary groups) and must NOT pass `--cpu`: the gfx1151 ROCm wheels exist + and a stale device-less container was found restarted 9,146 times. [[ComfyUI]] +- SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its ports off the gluetun + service. [[SearXNG]] [[Gluetun]] +- **Dankclaw's kernel command line is a repo file.** It boots **`iommu=pt`** since 2026-10-18, + tracked in `dankclaw/host/grub`, and moves ONLY through `scripts/set-dankclaw-bootline.sh` + (diff, install, `update-grub`, verify `grub.cfg`, `--reboot`) — never by hand-editing + `/etc/default/grub`. It replaced `amd_iommu=off`, and the local A/B on byte-identical traffic + says identity mode costs nothing, so the XDNA2 NPU returned for free. + [[dankclaw-iommu-translation-off]] +- **Rootless Quadlet containers run with `--replace --rm`**: `podman logs` covers only the CURRENT + instance; history across recreations lives in `journalctl --user -u .service`. Read + rootless storage WITHOUT sudo — it is per-user storage, so `sudo podman logs` finds nothing. + [[Quadlet Lifecycle]] +- **A dead rootless podman pause process breaks ALL container management on the host, and + `podman system migrate` panics in 5.7.0.** Every `podman` command fails with + `invalid internal status`, every start fails binding the bridges + (`failed to bind udp listener on 10.89.x.1:53`), every `Restart=always` quadlet loops — while + container storage stays intact. What is gone is `/run/user//libpod/tmp/alive` (a UUID + token) and the rootless netns; deleting the stale token is not enough. Repair: **reboot** + (`/run/user` clears and the quadlets return on their own — check `Linger=yes` and + `WantedBy=default.target` first). The 2026-09-16 occurrence had no OOM in `journalctl -k`: + treat the pause process as host state that can die for its own reasons. [[Quadlet Lifecycle]] +- **A container cannot reach a `Network=host` service at the host's LAN IP**: netavark DNATs + *published* ports only, so the host bind and the `10.89.1.1` gateway are both refused from a + bridge container while the published port answers. Use `host.containers.internal` + (169.254.1.2). This 502'd `hermes.franksdank.cloud` until its Caddyfile upstream changed. + [[Container Host Reachability]] +- **A `*.gguf` filter misses mradermacher's payloads: large quants ship as `.partNofM` + fragments** while the smaller rungs are single files, so filtering `endswith(".gguf")` makes a + whole quant class look absent. Fetch the parts, verify each against the Hub's LFS sha256, + concatenate in numeric order. [[Model Swap and Rollback]] +- **A fetch script must `-f` and verify, or it reports success on an access error.** + `curl -o FILE URL` exits 0 on HTTP 401/403 and writes the error body as the "artifact": a gated + repo yielded three 153-byte shards with identical sha256 and a cheerful `DOWNLOAD_DONE` in 20 + seconds for 104 GiB. Use `-f`/`--fail-with-body`, compare each file's byte count to the Hub's + declared size, and check the LFS sha256 (`scripts/download-frank-medium-gguf.sh` does all three; + ad-hoc fetches tend not to). Gated repos 403 even with the host's HF token until that account + accepts the gate. [[Model Swap and Rollback]] +- **A scout verdict is a lead, never a finding.** In the 2026-10-19 vault de-dup campaign six + read-only scouts worked the same pages: one classified era pages as stale *because* they + document retired payloads (that is an era page's content), one reported files that do not exist, + one reported duplications that grep disproved (`457.9 t/s` and the ping-RTT table are solo-held + by `Ethernet Bandwidth Math`), one cited a line number past the end of the file it judged, and + one drafted an Eras table with invented windows, payloads and rollback paths. Re-verify every + claimed duplicate, every claimed staleness and every quoted holder against the file and today's + fleet state before editing; a scout yield buys no evidence. ## Runbook Index diff --git a/scripts/check-okf.py b/scripts/check-okf.py new file mode 100755 index 0000000..f14dfa6 --- /dev/null +++ b/scripts/check-okf.py @@ -0,0 +1,175 @@ +#!/usr/bin/env python3 +"""OKF v0.1 conformance linter for the docs vault (serverAppConfigs-docs). + +The vault is an Open Knowledge Format bundle (omp-docs/references/okf-spec.md) +plus the maintenance discipline in okf-maintenance.md, plus this repo's own +index law (AGENTS.md -> *Documentation Policy*): an index is a locator, one row, +one clause, no numbers. Enforced here so a violation cannot be merged: + + FRONTMATTER every non-reserved page has a parseable frontmatter block with a + non-empty, non-generic `type` (the spec's only hard rule) + DESCRIPTION every concept has a description that is not the title, is not a + truncation artifact (ends in an ellipsis), carries no digits and + fits the one-clause budget (<= 120 chars) + TIMESTAMP every concept has an ISO `timestamp` + INDEX-FM an index.md carries no frontmatter (it is a listing, not a concept) + INDEX-COVER every index.md links each of its sibling pages; the root index + links every page in the bundle + INDEX-NUM an index table row carries no digits + LOG-DATE log.md date headings are `## YYYY-MM-DD`, newest first + LENGTH no page exceeds 500 lines (under-split); body under 15 lines is + flagged as an over-split stub + +Exit status is 1 when anything fails, so it gates a test. +""" +import datetime +import pathlib +import re +import sys + +RESERVED = {"index.md", "log.md"} +GENERIC_TYPES = {"file", "document", "page", "note"} +DESC_MAX = 120 +BODY_MIN = 15 +LINE_MAX = 500 + + +def split_frontmatter(text): + """Return (frontmatter_raw_or_None, body).""" + if not text.startswith("---"): + return None, text + end = text.find("\n---", 3) + if end == -1: + return None, text + return text[3:end].strip("\n"), text[end + 4:] + + +def field(raw, key): + if raw is None: + return None + m = re.search(rf"^{key}:\s*(.*)$", raw, re.M) + return m.group(1).strip() if m else None + + +def is_iso_date(value): + v = value.strip().strip("\"'").split("+")[0].split("T")[0] + try: + datetime.date.fromisoformat(v) + return True + except ValueError: + return False + + +def vault_pages(vault): + return [p for p in sorted(vault.rglob("*.md")) + if ".AGENTTRASH" not in str(p) and ".obsidian" not in str(p)] + + +def concept_pages(vault): + return [p for p in vault_pages(vault) if p.name not in RESERVED] + + +def check_concepts(vault, fail): + for page in concept_pages(vault): + rel = str(page.relative_to(vault)) + raw, body = split_frontmatter(page.read_text(errors="replace")) + if raw is None: + fail("FRONTMATTER", rel, "no frontmatter block") + continue + ty = field(raw, "type") + if not ty: + fail("FRONTMATTER", rel, "missing type") + elif ty.lower() in GENERIC_TYPES: + fail("FRONTMATTER", rel, f"generic type {ty!r}") + desc = (field(raw, "description") or "").strip().strip("\"'") + title = (field(raw, "title") or "").strip().strip("\"'") + if not desc or desc in ("-", "—"): + fail("DESCRIPTION", rel, "missing description") + elif desc == title: + fail("DESCRIPTION", rel, "description copies the title") + elif desc.endswith("…"): + fail("DESCRIPTION", rel, "description is a truncation artifact") + elif len(desc) > DESC_MAX: + fail("DESCRIPTION", rel, f"description is {len(desc)} chars (budget {DESC_MAX})") + elif re.search(r"\d", desc): + fail("DESCRIPTION", rel, "description carries numbers (index rows must not)") + ts = field(raw, "timestamp") + if not ts: + fail("TIMESTAMP", rel, "missing timestamp") + elif not is_iso_date(ts): + fail("TIMESTAMP", rel, f"unparseable timestamp {ts!r}") + lines = page.read_text(errors="replace").splitlines() + if len(lines) > LINE_MAX: + fail("LENGTH", rel, f"{len(lines)} lines (max {LINE_MAX})") + elif len([l for l in body.splitlines() if l.strip()]) < BODY_MIN: + fail("LENGTH", rel, "stub: fewer than %d body lines" % BODY_MIN) + + +def check_indexes(vault, fail): + root_index = vault / "index.md" + all_stems = {p.stem for p in concept_pages(vault)} + for index in sorted(vault.rglob("index.md")): + if ".AGENTTRASH" in str(index): + continue + rel = str(index.relative_to(vault)) + text = index.read_text(errors="replace") + if text.startswith("---"): + fail("INDEX-FM", rel, "index carries frontmatter") + siblings = {p.stem for p in index.parent.glob("*.md") if p.name not in RESERVED} + for stem in sorted(siblings): + if f"[[{stem}]]" not in text: + fail("INDEX-COVER", rel, f"does not link sibling {stem}") + if index == root_index: + for stem in sorted(all_stems - {p.stem for p in index.parent.glob("*.md")}): + if f"[[{stem}]]" not in text: + fail("INDEX-COVER", rel, f"does not link {stem}") + for line in text.splitlines(): + if line.startswith("|") and not re.match(r"^\|[\s-|]+$", line) and "Page" not in line: + if re.search(r"\d", line): + fail("INDEX-NUM", rel, f"row carries numbers: {line[:70]}") + + +def check_log(vault, fail): + for log in sorted(vault.rglob("log.md")): + if ".AGENTTRASH" in str(log): + continue + rel = str(log.relative_to(vault)) + dates = [] + for line in log.read_text(errors="replace").splitlines(): + m = re.match(r"^##\s+(.+)$", line) + if not m: + continue + head = m.group(1).strip() + dm = re.match(r"^(\d{4}-\d{2}-\d{2})$", head) + if not dm: + fail("LOG-DATE", rel, f"heading {head!r} is not a bare YYYY-MM-DD") + continue + if not is_iso_date(dm.group(1)): + fail("LOG-DATE", rel, f"heading {head!r} is not a real date") + dates.append(dm.group(1)) + if dates != sorted(dates, reverse=True): + bad = next(i for i in range(1, len(dates)) if dates[i] > dates[i - 1]) + fail("LOG-DATE", rel, f"{dates[bad]} after {dates[bad - 1]}: not newest first") + + +def main(argv): + vault = pathlib.Path(argv[1] if len(argv) > 1 else "serverAppConfigs-docs") + if not vault.is_dir(): + print(f"not a directory: {vault}") + return 2 + failures = [] + + def fail(rule, page, msg): + failures.append(f"{rule:12} {page}: {msg}") + + check_concepts(vault, fail) + check_indexes(vault, fail) + check_log(vault, fail) + print(f"{len(concept_pages(vault))} concept pages, {len(failures)} OKF violations") + for line in failures: + print(" " + line) + return 1 if failures else 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/scripts/gen-vault-index.py b/scripts/gen-vault-index.py new file mode 100755 index 0000000..e68bfdf --- /dev/null +++ b/scripts/gen-vault-index.py @@ -0,0 +1,135 @@ +#!/usr/bin/env python3 +"""Regenerate serverAppConfigs-docs/index.md from page frontmatter. + +The root index is a locator (AGENTS.md -> *Documentation Policy*): one row per +page, one clause, no numbers. Hand-maintaining it is how rows go missing, and +the docs seam fails when a page is not linked from it, so it is generated: + + * preamble everything above the first `## ` in the existing file is kept + verbatim (that is the curated statement of the vault's law) + * section from the page's directory, overridden by its frontmatter tags + (`endpoint` / `retired-endpoint` split the LLM pages) + * row text the page's frontmatter `description`, verbatim + +A page with no description is a hard error rather than an empty cell, so the +generator cannot silently produce a silent index. Sections with no pages are +skipped. Deterministic: the same vault always yields the same bytes. +""" +import pathlib +import re +import sys + +RESERVED = {"index.md", "log.md"} +SECTIONS = [ + ("Inference endpoints", "endpoint"), + ("Retired endpoints", "retired-endpoint"), +] +DIRECTORY_SECTIONS = [ + ("Dankclaw", "Dankclaw services"), + ("Dankserver", "Dankserver services"), + ("FrankBook-Pro", "FrankBook-Pro"), + ("Desktop", "Workstations"), + ("Infrastructure", "Cross-host infrastructure"), + ("Architecture", "Repo architecture and policy"), + ("Operations", "Operations and incidents"), + ("Agent", "Agent / harness"), + ("Research", "Research and methodology"), +] +PIPELINE_SECTION = "Pipeline Parallelism (series)" +LLM_FALLBACK = "LLM technique and hardware references" +CHANGE_SECTION = "Change history" +ORDER = [t for t, _ in SECTIONS] + [t for _, t in DIRECTORY_SECTIONS] + \ + [PIPELINE_SECTION, LLM_FALLBACK, CHANGE_SECTION] + + +def frontmatter_fields(text): + if not text.startswith("---"): + return {} + end = text.find("\n---", 3) + if end == -1: + return {} + raw = text[3:end] + fields = {} + for key in ("description", "type"): + m = re.search(rf"^{key}:\s*(.*)$", raw, re.M) + if m: + fields[key] = m.group(1).strip().strip("\"'") + tags = [] + m = re.search(r"^tags:\s*\[(.*?)\]$", raw, re.M) + if m: + tags = [t.strip().strip("\"'") for t in m.group(1).split(",") if t.strip()] + elif re.search(r"^tags:\s*$", raw, re.M): + block = re.search(r"^tags:\s*\n((?:\s+-\s+.*\n?)+)", raw, re.M) + if block: + tags = [l.strip().lstrip("-").strip().strip("\"'") + for l in block.group(1).splitlines() if l.strip()] + fields["tags"] = tags + return fields + + +def section_for(page, vault, fields): + rel = page.relative_to(vault) + if len(rel.parts) == 1: + return CHANGE_SECTION if page.name == "log.md" else "Uncategorised" + top = rel.parts[0] + if top == "LLM": + if PIPELINE_SECTION_DIR in rel.parts: + return PIPELINE_SECTION + for title, tag in SECTIONS: + if tag in fields.get("tags", []): + return title + return LLM_FALLBACK + for prefix, title in DIRECTORY_SECTIONS: + if top == prefix: + return title + return "Uncategorised" + + +PIPELINE_SECTION_DIR = "Pipeline Parallelism" + + +def main(argv): + vault = pathlib.Path(argv[1] if len(argv) > 1 else "serverAppConfigs-docs") + index = vault / "index.md" + if not index.is_file(): + print(f"missing {index}") + return 2 + existing = index.read_text() + preamble = existing.split("\n## ", 1)[0].rstrip("\n") + + pages = [p for p in vault.rglob("*.md") + if ".AGENTTRASH" not in str(p) and ".obsidian" not in str(p) + and p.name != "index.md" and p != index] + buckets, errors = {}, [] + for page in pages: + fields = frontmatter_fields(page.read_text(errors="replace")) + if page.name != "log.md": + desc = fields.get("description") + if not desc: + errors.append(str(page.relative_to(vault))) + continue + else: + desc = "Chronological incident and law log for the whole bundle." + buckets.setdefault(section_for(page, vault, fields), []).append((page.stem, desc)) + if errors: + print("pages with no description (fix the frontmatter, not the index):") + for e in sorted(errors): + print(" " + e) + return 1 + + out = [preamble, ""] + for title in ORDER + sorted(set(buckets) - set(ORDER)): + rows = sorted(buckets.get(title, []), key=lambda r: r[0].lower()) + if not rows: + continue + out += [f"## {title}", "", "| Page | Owns |", "|---|---|"] + out += [f"| [[{stem}]] | {desc} |" for stem, desc in rows] + out.append("") + out.append("") + index.write_text("\n".join(out) + "\n") + print(f"{index}: {sum(len(v) for v in buckets.values())} rows in {len([t for t in ORDER if buckets.get(t)])} sections") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/serverAppConfigs-docs/LLM/frank-zealot.md b/serverAppConfigs-docs/LLM/frank-zealot.md index 5028636..2b87766 100644 --- a/serverAppConfigs-docs/LLM/frank-zealot.md +++ b/serverAppConfigs-docs/LLM/frank-zealot.md @@ -98,7 +98,7 @@ re-run the same arms with the box quiet before a figure is carried anywhere else | Window | Payload | Runtime | Shape | Replaced because | Rollback | |---|---|---|---|---|---| | 2026-10-18 13:19 → same evening (the boot `68afc6d` replaced) | the same 9B MTP file and projector | PrismML fork 10777 | 3 × 262,144 (pool 786,432), q8_0 K+V = 13,056 MiB, settled 17,956 / 6,169 MiB; 168.7 t/s decode (three runs 168.4–168.7), ~300 t/s over three streams (102.8 each), 2,877 t/s prefill at 106,418 | Frank ordered unquantized KV, and that pool cannot survive it: 24,576 MiB of f16 cache > 24,123 addressable | one env line — `CTX_SIZE=786432 PARALLEL=3 CACHE_TYPE_K/V=q8_0` — and redeploy; no payload move | -| 2026-10-18, 24 min (arm under the unit) | `Bonsai2-PTQ1_0/Ternary-Bonsai-2-27B-PTQ1_0.gguf` 5,946,648,928 B + hardlinked Q8 mmproj | PrismML fork 10777 | 2 × 200,704, batch/ubatch 3072/1536 | Frank scratched the 27B plan — ~48 t/s decode vs 76.6 for PQ2_0 on the same weights | `/home/dank/models/Bonsai2-PTQ1_0/` + `scripts/download-frank-zealot-bonsai2-ptq1.sh` | +| 2026-10-18 03:14 → 13:19 (arm under the unit, then the leftover drop-in) | `Bonsai2-PTQ1_0/Ternary-Bonsai-2-27B-PTQ1_0.gguf` 5,946,648,928 B + hardlinked Q8 mmproj | PrismML fork 10777 | arm: 2 × 200,704, batch/ubatch 3072/1536. The leftover `frank-zealot.service.d/ptq1.conf` then made the 13:16 cutover deploy boot THIS payload at 3 × 133,888 (`new slot, n_ctx = 133888` x3, journal read 2026-10-19) until the 13:19 boot | Frank scratched the 27B plan — ~48 t/s decode vs 76.6 for PQ2_0 on the same weights | `/home/dank/models/Bonsai2-PTQ1_0/` + `scripts/download-frank-zealot-bonsai2-ptq1.sh` | | 2026-10-17 → 10-18 | Ternary-Bonsai-2-**27B** PQ2_0 7,206,168,928 B `3907dc16…` + Q8_0 mmproj 629,246,976 B | PrismML fork **10754** (`~/GitPackages/llama-prism/`) | 2 × 200,704 (pool 401,408), q8_0 K+V = 13,328 MiB, settled 22,664 / 1,461; 76.6 t/s decode, 109.4 aggregate over two streams, 691.7 t/s prefill at 166,027; MTP impossible — no head in the file | the 9B bills half the KV per token (8 full layers, not 16), which funded three slots at the native window and beat it on every rate | `/home/dank/models/Bonsai2-PQ2_0/` + the 10754 fork tree, `scripts/download-frank-zealot-bonsai2.sh`, revert the swap commit | | 2026-10-07 → 10-17 | Qwen3.5-9B-Uncensored-HauhauCS-Aggressive Q6_K 7,359,259,008 B `c0ba7beb…` + BF16 mmproj 921,704,448 B | llama.cpp master `5fc4f3c` CUDA sm_86 (`~/GitPackages/llama-master/build-cuda/`) | 2 × 200,704, exact f16 K+V, MTP off (no `nextn` tensors) | the ternary 27B bought a larger model class at the same window | `/home/dank/models/Qwen35-9B-HauhauCS-Q6K/` + the master build, `scripts/download-frank-zealot.sh` | | 2026-10-07 (same day) | Gemma4-26B-A4B-QAT Q4_K_M 16,796,015,520 B + BF16 mmproj 1,194,827,776 B + unused `gemma4-assistant` MTP 251,937,728 B | llama.cpp master CUDA | 1 × 200,704, f16 K + q8_0 V, ubatch 1024, MTP off; 21,868 / 2,257 settled, 139.9 t/s decode (the ~4 B-active MoE is at home here); MTP ON would cost ~1.9 GiB and boot at 319 MiB free | ONE slot serialized compaction; the 9B's lighter weights bought two slots and exact KV | `/home/dank/models/Gemma4-26B-A4B/` (ungated — plain `curl -fL`) | @@ -129,10 +129,12 @@ re-run the same arms with the box quiet before a figure is carried anywhere else 3 cells, and an armed depth-2 drafter triples it to 452.25), compute buffers at graph capture, the mmproj after the fit. On gemma4-class hybrids `attention_k_eq_v` does **not** shrink the allocation — the cache reserves the K/V pair twice, 6.125 B/elem at f16 K + q8_0 V. -- **`--flash-attn on` is pinned, not decorative**: a quantized V exists only on the FA path and the engine falls back silently without it. llama.cpp has **no true fp8 KV dtype** — q8_0 is the 8-bit class (1.0625 B/elem, ~8.5 effective bits). `GGML_CUDA_FA_ALL_QUANTS` is vestigial and build-specific: the PrismML fork carries it ON (since 2026-09-17) while the master build had it OFF and still dispatched Q4/Q5 KV unconditionally (`ggml-cuda/fattn.cu` has no guard), so quantized KV never required it. +- **`--flash-attn on` is pinned, not decorative**: a quantized V exists only on the FA path and the engine falls back silently without it. llama.cpp has **no true fp8 KV dtype** — q8_0 is the 8-bit class (1.0625 B/elem, ~8.5 effective bits). `GGML_CUDA_FA_ALL_QUANTS` is **deprecated upstream and inert in both trees on this host** (no reference anywhere in `ggml/src/ggml-cuda/`, read 2026-10-19): master's `ggml/cmake/common.cmake` warns `use GGML_CUDA_FA_QUANTS=all instead`, and the successor's default — `GGML_CUDA_FA_QUANTS="q4_0-q4_0;q8_0-q8_0;f16-f16;bf16-bf16"` (`ggml/CMakeLists.txt:208`) — already compiles the quantized-KV kernels with the old flag OFF (master's `build-cuda` cache says `FA_ALL_QUANTS:BOOL=OFF` and it dispatched q8_0/q4_0 KV regardless). Prove a build's quantized-KV ability from its `CMakeCache.txt` `GGML_CUDA_FA_QUANTS` line, never from the retired flag or the build command you remember. - **Prices come from boots, not formulas, and an arm belongs to the payload it was measured on**: 3 × 180,224 + - q8_0 V fits with Q5_K_M (21,684 used / 2,441 free) and does **not** fit with Q6_K inside the 2–3 GiB band. The - mmproj worst case (~870 MiB) is printed before fitting but never billed by the engine's own fit check. Only + q8_0 V fits with Q5_K_M (21,684 used / 2,441 free) and does **not** fit with Q6_K inside the 2–3 GiB band. + The mmproj worst case is printed before fitting (`estimated worst-case memory usage of mmproj is + 873.10 MiB` on the Bonsai-2 27B boot) but never billed by the engine's own fit check: that boot + projected **21,439 MiB** and settled at **22,664**. Only `tests/frank_zealot_unit.bats`'s fit gate can see a boot that would die at `cudaMalloc`, so every retune keeps it green — and the first verbosity-4 ledger plus settled `nvidia-smi` of a new dtype belongs on this page, because projections drift. @@ -179,7 +181,10 @@ re-run the same arms with the box quiet before a figure is carried anywhere else `predicted_per_second`). Keep `VERBOSITY=4` — 3 is the engine default and cannot prove the KV or draft sizes, while levels above 4 flood the journal. - llama.cpp does **not** validate the requested model id: it answers and echoes the served `ALIAS`, so a rename breaks id-keyed consumers (the OpenWebUI `model` row, its `access_grant`, `openai.api_configs`) and nothing server-side. Alias history: `frank-small` → `frank-secondary` (2026-10-06) → `frank-zealot` (`7c77326`, 2026-10-09); the service and unit took the name 2026-10-10 (`55f87fc`). -- The first request after a boot is a warmup artifact (37 t/s class) and is never a measurement; never time a wall-clock aggregate on a generation that stops on its own (use `ignore_eos`). +- The first request after a boot is a warmup artifact, never a measurement: on the Bonsai-2 27B + boot request #1 gave 2.6 t/s prefill / 37.7 t/s decode and request #2 gave 1,258 t/s prefill / + 54.7 t/s per stream (two concurrent, the 109.4 aggregate in the Eras row) — a 3x spread on one + shape. Never time a wall-clock aggregate on a generation that stops on its own (use `ignore_eos`). ## Files