dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(agents): compress Key Lessons to one law per bullet, correct the FA-quants law

Browse Source 
Key Lessons Learned 483 -> 340 lines. Every law survives; only the essays
went. Port-before-cut census over the diff: 454 deleted lines, 0 orphaned
facts (every number still has a holder in the vault).

- Each lesson is now one bullet with the page that owns the detail.
- De-duplicated the page's own internal repeats: the layer-split law was
  told twice (PCIe and ROCm sections), the FA-quants law twice more.
- CORRECTION, verified on dankclaw 2026-10-19: `GGML_CUDA_FA_ALL_QUANTS` is
  deprecated upstream (master's ggml/cmake/common.cmake tells you to use
  GGML_CUDA_FA_QUANTS=all), the successor's default pair list
  `q4_0-q4_0;q8_0-q8_0;f16-f16;bf16-bf16` compiles the quantized-KV kernels
  with the old flag OFF (master build-cuda cache: FA_ALL_QUANTS=OFF), and
  neither llama tree references the old macro under ggml/src/ggml-cuda/.
  The "required or prefill collapses ~12x" law is obsolete; the new law is
  to read the CMakeCache pair list.
- frank-zealot: ported the three facts AGENTS.md alone carried, and verified
  them against the host first — the 2026-10-18 13:16 cutover boot really did
  load PTQ1_0 at `new slot, n_ctx = 133888` x3 (journal), the mmproj fit
  check really printed 873.10 MiB against a 21,439 MiB projection, and the
  warmup spread is 2.6 -> 1,258 t/s prefill.
- New tooling: scripts/check-okf.py (OKF v0.1 conformance linter) and
  scripts/gen-vault-index.py (root index regenerated from frontmatter).

Co-Authored-By: omp <omq@openclaw.sh>
This commit is contained in:
Frank
2026-10-19 02:51:32 -07:00
co-authored by omp
parent a5fe8db877
commit 1ed7f0ddf2
4 changed files with 631 additions and 459 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+311 -454
View File
@@ -475,481 +475,338 @@ linked runbooks.
**PCIe / multi-GPU**
- **RCCL collectives need PCIe atomics on every participating GPU.** RCCL's
kernels carry a hidden hostcall-buffer parameter; ROCr refuses to dispatch
them when the device has no atomics, and the failure surfaces as
`hipErrorIllegalState` ("the operation cannot be performed in the present
state") from `hipLaunchKernel`, not as anything that names atomics. The
value is consumed at driver probe time, so writing the AtomicOp Requester
bit with `setpci` at runtime changes nothing. Check the *chain*:
`lspci -vv -s <gpu>` and every upstream bridge must show
`AtomicOpsCap: Routing+` (a `Routing-` bridge is a hardware wall), and the
endpoint's Device Control 2 (`setpci -s <gpu> CAP_EXP+28.w`) must have bit
6 set. AMD desktop platforms put one card on CPU root ports (atomics work)
and the rest behind the chipset (usually not) — that asymmetry is what
blocked frank-medium's TP=2. A one-card collective proves nothing: NCCL
short-circuits at `world_size=1`.
- **That wall was physical, and it was fixed on 2026-09-20 by a board swap.**
Both paths that put two cards on CPU root ports were: an M.2→PCIe x4 riser
into a *CPU-attached* M.2 slot, or a board whose CPU lanes bifurcate to
x8/x8. dankserver took the second: **ASUS ProArt B850-Creator WiFi NEO**,
`PCIEX16_1` + `PCIEX16_2` at x8/x8 Gen5 on root ports `00:01.1` / `00:01.3`,
every bridge now `Routing+` and **both endpoints `ReqEn+`** — the exact
precondition the old test plan named. Bandwidth was never the constraint (a
27B TP=2 allreduce at batch 4 is ≈10 MB per token-step, ≈1.4 ms at Gen4 x4
and under 0.4 ms at Gen5 x8).
**Proven on the same day.** vLLM's R4D one-shot allreduce runs TP=2 here and
frank-medium serves on it; llama.cpp's RCCL path was built too
(`GGML_HIP_RCCL=ON`) and its tensor mode measured — real collective, slower
shape. Note the collective choice matters: TheRock's RCCL builds lack gfx1201
device code (`invalid device function`), which is why R4D is the path vLLM
uses at all. Evidence matrix and the measured three-arm verdict:
[[Dual-R9700-Cooperation]].
- **When a GPU collective dies, read the runtime, not the framework.** The
message that identified the cause came from `AMD_LOG_LEVEL=3`
(`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch
failed!`), not from vLLM or RCCL. `AMD_LOG_LEVEL` costs one env var and
prints the failing HIP call with its return code.
- **The software path that puts both cards on ONE model without a collective:
llama.cpp's layer split.** On the old board vLLM could not do it in any of its
parallelisms — TP needs an allreduce, PP needs the same p2p primitive, and both
died on the card that could not dispatch an RCCL kernel. `--split-mode
layer` in llama.cpp needs no
collective: layers are distributed in order and only the boundary activation
crosses PCIe, once per forward pass. Measured here: 1991 t/s prefill /
18.8 tok/s decode layer-split, against 272 / 26.3 tensor (butterfly
fallback).
This was what `frank-medium` ran until 2026-09-20, when the board swap made
collectives legal and radiance TP=2 beat it on every axis (95.8 vs 45.7 t/s
decode, 0.21 vs 0.72 s TTFT, 259 vs 72 t/s burst-4) — see
[[frank-medium-radiance-mxfp4]]; the llama shape is the rollback target
[[frank-medium-llamacpp-dflash2]].
- **Layer split is also the cross-vendor answer — and `-ot` is what cannot be
combined with it.** On Dankclaw (2026-09-16) a CUDA 3090 + Vulkan iGPU layer
split ran a real two-device graph: `-dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85`
held 14,994 MiB on the 3090 and gave 27.2 tok/s with 268.5 t/s prefill at 64k
depth (iGPU alone: 25.8 / 231.8). What does NOT work is per-tensor placement
across devices: `--override-tensor <group>=<device>` sets
`has_tensor_overrides()`, a REQUIRED term of the scheduler's
`pipeline_parallel` flag (llama-context.cpp:433 — the only place in the tree
that reads it), and that flag is what sets `n_copies` in
`ggml_backend_sched_new`; with `n_copies == 1` a pre-allocated weight on a
non-primary device has no backend and the graph aborts (`ggml-backend.cpp:941`,
reported with the leaf's own op name `(NONE)`). `=CPU` is the only override
target that works, because the CPU backend is last in the list. Pure Vulkan
changes nothing here — the gate is the override, not the driver mix. See
[[x4-pair-experiment]].
- **A host↔3090 link at x4 measures 6.72 GB/s, and that is a design
constraint, not a detail.** `pcie-link-test.cu` on Dankclaw (2026-09-16,
services stopped): pinned H2D 6.72 GB/s, D2H 6.60 GB/s — 85 % of the Gen4 x4
ceiling — with async no better than sync and pageable within 1 % of pinned,
i.e. the copies are link-limited rather than staging-limited (the usual
"pageable costs 2x" rule does not apply here). The card is x16-capable and the
Strix Halo port beneath it is x8-capable (`LnkCap: 16GT/s, Width x8` on
`00:03.1`), so the x4 lives in the adapter or cable; the link idles at
2.5 GT/s x4 and trains up under load. Consequence for any engine that
*streams* weights across it: a 3 MiB slab costs 474 us, and a 111 GiB A3B
model needs ~1.70 GiB of active experts per token ⇒ **3.7 tok/s** (4.6 with a
20 % resident cache), against **20.46 tok/s** for the same model with the
experts *computed* in host RAM by llama.cpp (`-ot exps=CPU`, whence the link
carries activations only) and 25–26.5 on the iGPU. Streaming over PCIe loses;
computing in RAM does not care. FreeToken — expert LRU in host RAM, exactly
this streaming shape — also loads only modelopt block-FP8 safetensors for
`qwen3_5_moe`, so a Q4_K_M GGUF is unloadable there before bandwidth is even
the question. See [[x4-pair-experiment]].
- **RCCL collectives need PCIe atomics on every participating GPU, and the failure never names
atomics.** The kernels carry a hidden hostcall-buffer parameter; ROCr refuses to dispatch them on
a device without atomics and the error surfaces as `hipErrorIllegalState` from `hipLaunchKernel`,
not as anything that names atomics. The value is consumed at driver PROBE time, so writing the
AtomicOp Requester bit with `setpci` at runtime changes nothing. Check the CHAIN: `lspci -vv -s
<gpu>` and every upstream bridge must show `AtomicOpsCap: Routing+` (a `Routing-` bridge is a
hardware wall), and the endpoint's Device Control 2 (`setpci -s <gpu> CAP_EXP+28.w`) must have bit
6 set. AMD desktop platforms put one card on CPU root ports (atomics work) and the rest behind the
chipset (usually not) — that asymmetry is what blocked TP=2 here. A one-card collective proves
nothing: NCCL short-circuits at `world_size=1`. [[PCIe Link and Atomics]]
- **That wall was physical, and a board swap removed it on 2026-09-20.** dankserver took the path of
a board whose CPU lanes bifurcate to x8/x8 Gen5, both cards on root ports, every bridge `Routing+`
and BOTH endpoints `ReqEn+` — the exact precondition the old test plan named. Bandwidth was never
the constraint (a 27B TP=2 allreduce at batch 4 is ≈10 MB per token-step, under 0.4 ms at Gen5 x8).
The collective choice matters: TheRock's RCCL builds lack gfx1201 device code
(`invalid device function`), which is why vLLM runs its R4D one-shot allreduce here at all.
Evidence matrix and the measured three-arm verdict: [[Dual-R9700-Cooperation]].
- **When a GPU collective dies, read the runtime, not the framework.** The message that identified
the cause came from `AMD_LOG_LEVEL=3` (`Pcie atomics not enabled, hostcall not supported` →
`AQL dispatch failed!`), never from vLLM or RCCL — the env var costs one line and prints the
failing HIP call with its return code. [[PCIe Link and Atomics]]
- **Layer split is the no-collective path, on one vendor's cards and across two — and
`--override-tensor` is what cannot ride with it.** Layers go in order and only the boundary
activation crosses PCIe once per forward pass, so it works wherever a collective cannot be
dispatched (and on an RCCL-less fork it beats tensor mode, which falls back to a butterfly
allreduce); it is also the cross-vendor answer, where a CUDA + Vulkan split held a real
two-device graph. But `--override-tensor <group>=<device>` sets `has_tensor_overrides()`, a
REQUIRED term of the scheduler's `pipeline_parallel` flag (llama-context.cpp), and that flag sets
`n_copies` in `ggml_backend_sched_new`: with `n_copies == 1` a pre-allocated weight on a
non-primary device has no backend and the graph aborts (ggml-backend.cpp, reported with the leaf's
own op name `(NONE)`). `=CPU` is the only override target that works, because the CPU backend is
last in the list, and pure Vulkan changes nothing — the gate is the override, not the driver mix.
Pin the spec-decode drafter to one device: an unpinned draft under tensor mode aborts the KV build,
and on a split its home is the card holding the target's last layers.
[[Llama Cpp Multi-Device Splits]] [[x4-pair-experiment]]
- **A host↔3090 link at x4 measures 6.72 GB/s, and that is a design constraint, not a detail.**
Pinned H2D 6.72 / D2H 6.60 GB/s (≈85 % of the Gen4 x4 ceiling), async no better than sync and
pageable within 1 % of pinned — the copies are LINK-limited, not staging-limited, so the usual
"pageable costs 2×" rule does not apply. The card and the port beneath it are both wider than x4,
so the x4 lives in the adapter or cable, and the link idles at 2.5 GT/s until load. Consequence
for any engine that STREAMS weights across it: a 3 MiB slab costs 474 µs, so a 111 GiB A3B model
needing ~1.70 GiB of active experts per token lands at 3.7 tok/s, against 20.46 tok/s for the same
model with the experts COMPUTED in host RAM (`-ot exps=CPU`, where the link carries activations
only). Streaming over PCIe loses; computing in RAM does not care. [[x4-pair-experiment]]
**ROCm / GPU**
- **Layer split needs no collective** — layers go in order and only the boundary
activation crosses PCIe, so it is the fallback wherever a collective cannot be
dispatched (and on an RCCL-less fork it beats tensor mode, which falls back to
a butterfly allreduce). Pin the spec-decode drafter to one device: an unpinned
draft under tensor mode aborts the KV build, and on a split its home is the
card holding the target's last layers. Deployed history:
[[frank-medium-llamacpp-dflash2]].
- **Bundling a ROCm runtime next to a llama.cpp binary: `cp -r SRC DST/` is a
trap.** With a missing `DST`, cp creates `DST` as a copy of `SRC` and drops
one directory level. rocBLAS and hipBLASLt look for their device code at a
fixed depth (`../lib/rocblas/library`, `../lib/hipblaslt/library/<arch>`),
and the flattened copy fails at *request* time, not load time: the engine
reads the weights, answers `/health` and dies on the first completion with
`Cannot read .../TensileLibrary.dat: No such file or directory for GPU arch
: gfx1201`. Declare the destination, then copy the contents.
- **mmap must stay on** for the big-PLE models: `--no-mmap` forces the
~50 GiB PLE table into anonymous memory → silent cgroup kill, dmesg is the
only evidence. mmap pages ARE page cache. Never pin the PLE tensor with
`-ot` (23 → 13.4 t/s).
- **HIP reads bare env vars** `TEMP`, `HOST`, `GPU_MAX_HEAP_SIZE`. Serve
scripts export env files with `set -a`, so a tunable named `TEMP` leaks
into HIP (`TEMP=1.0` sent the JIT cache to path `1.0` and segfaulted every
load, fixed 2026-08-13 by renaming to `TEMPERATURE`). Never name a tunable
these; check with
- **Bundling a ROCm runtime next to a llama.cpp binary: `cp -r SRC DST/` is a trap.** With a missing
`DST`, cp creates `DST` as a copy of `SRC` and drops one directory level. rocBLAS and hipBLASLt
look for their device code at a FIXED depth (`../lib/rocblas/library`,
`../lib/hipblaslt/library/<arch>`), so the flattened copy fails at REQUEST time, not load time:
the engine reads the weights, answers `/health` and dies on the first completion with
`Cannot read .../TensileLibrary.dat: No such file or directory for GPU arch : gfx1201`. Declare
the destination, then copy the contents.
- **mmap must stay on** for the big-PLE models: `--no-mmap` forces the ~50 GiB PLE table into
anonymous memory → silent cgroup kill, with dmesg as the only evidence. mmap pages ARE page cache,
and pinning the PLE tensor with `-ot` cost 23 → 13.4 t/s. [[Llama Cpp Serving Sizing]]
- **HIP reads bare env vars** `TEMP`, `HOST`, `GPU_MAX_HEAP_SIZE`. Serve scripts export env files
with `set -a`, so a tunable named `TEMP` leaks into HIP (`TEMP=1.0` sent the JIT cache to path
`1.0` and segfaulted every load, fixed 2026-08-13 by renaming to `TEMPERATURE`). Never name a
tunable these; check with
`strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE"`.
- **`-DGGML_CUDA_FA_ALL_QUANTS=ON` is required** for q8_0 KV + flash
attention on CUDA builds; without it prefill collapses ~12x (95 vs 1300+
t/s, measured 2026-08-17).
- **Flash-Next MTP heads do not load on any current loader** — they need full
per-layer `blk.<il>.hc_attn_norm.weight` and ship only `blk.48` (verified
3x). Don't re-attempt without new loader support.
- **QSA flash-attn assert:** `GGML_ASSERT(a-type == GGML_TYPE_F32)` on
qwen4exp needs `scripts/patches/qwen4exp-qsa-f32-mask.patch` — but only on
the agentionai FAST layer set; the kingjones STRIX set works unpatched.
- **GPU budget on Dankclaw is the GTT aperture (~114 GiB), not the
rocm-smi "dedicated" figure.** The Radeon 8060S (gfx1151) reports
116,732 MiB (`llama-server --list-devices`;
`gtt_total=122402963456` in `/sys/class/drm/card1/device/`). The older
"64 GiB, SVM-backed" lesson was already stale — the CIRU Orca predecessor
held 92.6 GiB of weights on the iGPU. `free`/`dmidecode` show the
CPU-side carve-out — intentional, do not change.
- **`GGML_CUDA_FA_ALL_QUANTS` is dead; do not build for it.** Verified on this host 2026-10-19:
master's `ggml/cmake/common.cmake` warns it is deprecated in favour of `GGML_CUDA_FA_QUANTS`, the
successor's DEFAULT pair list (`q4_0-q4_0;q8_0-q8_0;f16-f16;bf16-bf16`) already compiles the
quantized-KV kernels with the old flag OFF, and neither llama tree on this host references the old
macro anywhere under `ggml/src/ggml-cuda/`. The old "~12× prefill collapse" belongs to the older
source line. Prove a build's KV ability from its `CMakeCache.txt`. [[frank-zealot]]
- **Flash-Next MTP heads do not load on any current loader** — they need full per-layer
`blk.<il>.hc_attn_norm.weight` and ship only `blk.48` (verified 3×). Do not re-attempt without new
loader support.
- **QSA flash-attn assert:** `GGML_ASSERT(a-type == GGML_TYPE_F32)` on qwen4exp needs
`scripts/patches/qwen4exp-qsa-f32-mask.patch` — but only on the agentionai FAST layer set; the
kingjones STRIX set works unpatched. [[Llama Cpp Serving Sizing]]
- **The GPU budget on Dankclaw is the GTT aperture, not the rocm-smi "dedicated" figure.** The
Radeon 8060S (gfx1151) reports 116,732 MiB (`llama-server --list-devices`; `gtt_total` in
`/sys/class/drm/card1/device/`), and the older "64 GiB, SVM-backed" lesson was already stale.
`free`/`dmidecode` show the CPU-side carve-out — intentional, do not change. [[Dankclaw System Specs]]
**Load / crash diagnosis**
- Model load is slow (minutes cold); health returns 503 while loading; do
not kill the process. `deploy.sh` waits up to 20 minutes (`LLM_WAIT`). On
halogen, the FIRST request after a (re)start can go silent ~45–60 s
(health `engine_unresponsive`, 503s) and then self-recovers — do not
restart over it.
- Crash-loop procedure: (1) `journalctl -u <svc> -n 100` + kernel line via
`journalctl -k` (segfault at address 8 in `libamdhip64` = NULL unref in
HIP); (2) run serve script, then raw binary — different outcome isolates
the env file; (3) `gdb -batch -ex run -ex "bt 25" --args bash
scripts/serve-<svc>.sh` — llama-server buffers stdout and loses the log
tail on SIGSEGV; trust gdb, not the log; (4) `amdgpu: SVM mapping failed`
spam is a symptom, not a root cause.
- Journal signatures: client abort logs `W srv stop: cancel task, id_task =
N`; natural stop logs only `slot release` with `truncated = 0`. The server
never preempts a running generation; excess tasks queue when slots are
busy.
- Model load is slow (minutes cold) and health returns 503 while it runs: do NOT kill the process,
`deploy.sh` waits up to 20 minutes (`LLM_WAIT`). On halogen the FIRST request after a (re)start
can go silent for ~45–60 s (`engine_unresponsive`, 503s) and then self-recovers — do not restart
over it. [[frank-dragoon]]
- Crash-loop procedure: (1) `journalctl -u <svc> -n 100` AND the kernel line via `journalctl -k` (a
segfault at address 8 in `libamdhip64` is a NULL unref in HIP); (2) run the serve script, then the
raw binary — a different outcome isolates the env file; (3) `gdb -batch -ex run -ex "bt 25"
--args bash scripts/serve-<svc>.sh`, because llama-server buffers stdout and loses the log tail on
SIGSEGV: trust gdb, not the log; (4) `amdgpu: SVM mapping failed` spam is a symptom, not a root
cause. [[LLM Performance Diagnosis]]
- Journal signatures: a client abort logs `W srv stop: cancel task, id_task = N` while a natural stop
logs only `slot release` with `truncated = 0`. The server never preempts a running generation;
excess tasks queue when slots are busy.
**Host memory / swap**
- Swap resize ordering: create and `swapon` the replacement BEFORE `swapoff`
of the old one, or live pages must land in RAM that `MemAvailable`
overstates. Dankclaw swap (64 GiB at `/swap.img`, fstab-driven) protects
the co-tenants, not frank-dragoon — pinned weights don't swap, and paging
the KV pool is worse than the OOM it prevents.
- Host swap state is invisible to `deploy.sh` (no repo file) — check
`swapon --show` before reasoning about headroom.
- **A sysfs raise is a countdown to the next reboot.** `ttm.pages_limit` defaults
to half of RAM, so the manual raise to 200 GiB (2026-10-05) vanished with the
2026-10-12 23:17 POWER FAILURE on dankserver: the cap fell to 125,988 MiB,
frank-archon's pinned-ledger preflight (floor 150,000 MiB) refused the start,
and `Restart=always` re-ran that refusal **6,919 times in ~20 h** — a unit in
`activating (auto-restart)` that exits 1 in 50 ms with no container and no
engine log is a PREFLIGHT refusal, not an engine fault. Repo law now:
`scripts/ttm-preflight.sh` (sourced by `serve-frank-archon.sh`) raises the knob
to `FRANK_ARCHON_TTM_TARGET_MIB` itself, re-reads, and refuses by name only
after a write the kernel rejected. Never prescribe a manual sysfs write as the
recovery procedure for a service that can boot before a human reaches it.
- **Silent RAM corruption outranks every software theory.** frank-archon's
~40k-token output corruption (2026-10-04/05, repetition/garbage in long
agent conversations) was marginal DDR5 at 4800 MT/s on four DIMMs, fixed
by 4600. Consumer non-ECC memory raises **no MCE** for a silent flip, and
a short in-band pattern check (60 s, 8 GiB of 246 GiB) is not a stability
proof — combined, those two instruments said "ruled out" while the RAM
was the cause. "4800 is the JEDEC floor on 5600-rated DIMMs" is a rating,
not a margin guarantee. When the fault is data-shaped (NaN states,
garbage tokens) and a software revert doesn't move it, price in the RAM
before the runtime — and remember a revert tested under the same marginal
hardware is itself confounded. See the runbook's 2026-10-05 section.
- **Swap resize ordering: create and `swapon` the replacement BEFORE `swapoff` of the old one**, or
live pages must land in RAM that `MemAvailable` overstates. Dankclaw's 64 GiB `/swap.img`
(fstab-driven) protects the co-tenants, not an endpoint with pinned weights — paging the KV pool
is worse than the OOM it prevents. Host swap state is invisible to `deploy.sh` (no repo file):
`swapon --show` before reasoning about headroom. [[Swap Configuration]]
- **A sysfs raise is a countdown to the next reboot.** `ttm.pages_limit` defaults to half of RAM, so
a manual raise vanished with a power failure, frank-archon's pinned-ledger preflight refused the
start, and `Restart=always` re-ran that refusal 6,919 times in ~20 h — **a unit in
`activating (auto-restart)` that exits 1 in 50 ms with no container and no engine log is a
PREFLIGHT refusal, not an engine fault.** Repo law now: `scripts/ttm-preflight.sh` raises the knob
to `FRANK_ARCHON_TTM_TARGET_MIB` itself, re-reads, and refuses by name only after a write the
kernel rejected. Never prescribe a manual sysfs write as the recovery procedure for a service that
can boot before a human reaches it. [[frank-archon]]
- **Silent RAM corruption outranks every software theory.** frank-archon's long-conversation
repetition/garbage was marginal DDR5 at 4800 MT/s on four DIMMs, fixed by dropping to 4600.
Consumer non-ECC memory raises NO MCE for a silent flip, and a short in-band pattern check
(60 s over 8 GiB of 246 GiB) is not a stability proof — combined, those two instruments said
"ruled out" while the RAM was the cause, and "4800 is the JEDEC floor" is a rating, not a margin
guarantee. When the fault is data-shaped (NaN states, garbage tokens) and a software revert does
not move it, price in the RAM before the runtime — and remember a revert tested under the same
marginal hardware is itself confounded. [[log]]
**Runtime-specific guards**
- **Rootless podman cannot RAISE rlimits, and `--ulimit nofile=1048576`
dies at container create, not at runtime.** runc reports
`error setting rlimit type 7: operation not permitted` (type 7 =
RLIMIT_NOFILE) when the requested hard limit exceeds the host process's
hard limit — on Dankclaw that ceiling is 524288. Inherited-equal limits
are fine, so `memlock=-1:-1` works iff the unit grants
`LimitMEMLOCK=infinity` (systemd is privileged; the value flows through
podman to runc without any raise). Upstream halogen's own run lines pass
NO `--ulimit` at all — copy vendor run lines before adding flags
(measured 2026-10-05 on the frank-dragoon cutover).
- **Kairic Edge** (if ever re-deployed): the pinned CK commit needs
`third_party/composable_kernel/include/ck/config.h` generated manually
(untracked; `git clean` removes it), and its greedy-argmax fastpath rejects
non-greedy requests and reasoning budgets. Re-test chat before production.
- **FastMTP sidecar**: build must be commit `4df29be4` + the HauhauCS patch;
stock binaries fail the draft load with a tensor-shape mismatch.
- **A GGUF's custom tensor type picks the runtime — read the histogram before
promising a file to a loader.** `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is
ftype 115 with 311 tensors of type 103 (`GGML_TYPE_Q8_0_ROCMFPX`); stock
llama.cpp and gufo refuse it at load, so ROCmFPX was not a preference but
the only door. Counting tensor types out of the GGUF KV section takes one
python read and settles a runtime argument in minutes. The build identity
matters as much as the fork: the 2026-08-25 ROCmFPX builds SIGSEGV in
`ggml_compute_forward_set_rows` on this same file, so
`scripts/build-rocmfpx.sh` pins commit `36e9acd` by SHA.
- **Speculative-decode flags move as a SET.** Handing the ROCmFPX fork
`--model-draft` with no `--spec-type` (the naive way to build a "no MTP"
bench arm) segfaulted at load, core dumped, no diagnostic (2026-10-02). A
baseline arm must strip `--model-draft` AND every `--spec-draft-*` AND
`--spec-type`, or it measures a crash instead of the absence of drafting.
- **An orphaned scratch server taxes the whole host.** A bench child left
alive by a client-side exception kept ~26 GiB of weights resident on a
121 GiB box, pushed `available` to 3 GiB, and every later rate was suspect.
Record the child pid and kill it in a `finally`, then prove `free` came back
before believing a number. Kill by pid — never `pkill -f llama-server`,
which also hits frank-zealot.
- **Rootless podman cannot RAISE rlimits, so `--ulimit nofile=1048576` dies at container CREATE**,
not at runtime: runc reports `error setting rlimit type 7: operation not permitted` (type 7 =
RLIMIT_NOFILE) when the requested hard limit exceeds the host process's ceiling (524288 on
Dankclaw). Inherited-equal limits are fine, so `memlock=-1:-1` works iff the unit grants
`LimitMEMLOCK=infinity` — systemd is privileged and the value flows through podman to runc without
any raise. Upstream halogen's own run lines pass NO `--ulimit` at all: copy vendor run lines
before adding flags. [[frank-dragoon]]
- **A GGUF's custom tensor type picks the runtime — read the histogram before promising a file to a
loader.** `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is ftype 115 over 311 tensors of type 103
(`GGML_TYPE_Q8_0_ROCMFPX`), so stock llama.cpp and gufo refuse it at load and the fork was not a
preference but the only door. Counting tensor types out of the GGUF KV section takes one python
read and settles a runtime argument in minutes. Build identity matters as much as the fork: the
2026-08-25 ROCmFPX builds SIGSEGV in `ggml_compute_forward_set_rows` on this same file, so
`scripts/build-rocmfpx.sh` pins commit `36e9acd` by SHA. [[GGUF Compatibility]]
- **Speculative-decode flags move as a SET.** Handing the fork `--model-draft` with no `--spec-type`
(the naive way to build a "no MTP" bench arm) segfaulted at load, core dumped, no diagnostic. A
baseline arm must strip `--model-draft` AND every `--spec-draft-*` AND `--spec-type`, or it
measures a crash instead of the absence of drafting. [[frank-zealot]]
- **An orphaned scratch server taxes the whole host.** A bench child left alive by a client-side
exception kept ~26 GiB of weights resident on a 121 GiB box and pushed `available` to 3 GiB, so
every later rate was suspect. Record the child pid, kill it in a `finally`, then prove `free` came
back before believing a number — and kill by pid, never `pkill -f llama-server`.
[[LLM Bench Windows and Measurement]]
- Retired-runtime traps that can come back: **Kairic Edge** needs
`third_party/composable_kernel/include/ck/config.h` generated manually (untracked, so `git clean`
removes it) and its greedy-argmax fastpath rejects non-greedy requests and reasoning budgets, so
re-test chat before production; the **FastMTP sidecar** must be commit `4df29be4` + the HauhauCS
patch, since stock binaries fail the draft load with a tensor-shape mismatch.
**llama.cpp serving**
- **MTP is a property of the FILE, not of the flags.** The `-mtp` GGUF build
carries the head as `blk.<n>.nextn.*` (`qwen35.nextn_predict_layers = 1`);
`--spec-type draft-mtp` then sets `load_mtp`, and the draft context runs on
the MAIN model's weights (`shares_model`), so there is no second copy of a
12 GiB model in VRAM and no `-md` argument. The plain build has no MTP
tensors at all, and pointing `--model-draft` at the file would load it twice.
- **What can be chained, what cannot, and what a chain costs on a hybrid.**
`--spec-type a,b` builds every enabled implementation and chains drafting
through them, in a fixed internal order (ngram types first, then the
drafters — CLI order is irrelevant). The gate is `ctx_dft`: **draftless**
types (the ngram family) are added unconditionally, while `draft-mtp`,
`draft-dflash`, `draft-dspark` and `draft-eagle3` all require a draft
CONTEXT. So a chain may hold one context-backed drafter plus any number of
ngram types — `draft-dflash,ngram-mod` is legal and measured (2.6x the
drafter alone on prompt-echo traffic, byte-identical to it on prose at the
fleet's 48/64 window) — but `draft-mtp,draft-dflash` dies at load: *"context
type MTP requested but model doesn't contain MTP layers"*, because MTP
builds its context against whatever model the draft context holds. Two
context-backed types cannot share one process, and MTP's head only loads
when MTP **is** the draft model. **A LEGAL chain can still lose on a
recurrent model:** `draft-mtp,ngram-mod` on frank-dragoon's qwen35 hybrid
measured −15 % prose / −7 % block-copy (2026-10-03) — adding the ngram layer
logs `recurrent state rollback is not compatible with 'ngram-mod' -
disabling rollback support`, and MTP verification depends on that rollback,
so acceptance drops on ALL traffic even where the matcher fires. Measure
chains per architecture, never by analogy.
- **A drafter's KV scales with the TARGET's context, and its window policy
decides the price.** The draft context is built at `llama_n_ctx(ctx_tgt)` with
no flag to cap it, so a drafter declaring NO sliding window caches the whole
window on `--spec-draft-device` and can OOM the first draft graph capture
AFTER `/health` has answered. Read `sliding_window` out of the drafter's GGUF
header before sizing, and prefer a skewed `-ts` over a context cut. Worked
example: [[frank-medium-llamacpp-dflash2]].
- **A draft context bills its own f16 KV over the WHOLE POOL, and on a
single-card endpoint that squeeze can cost more than the drafter wins.** With
`--parallel N` the drafter pays the pool, not the slot, plus its own compute
buffers. So the MTP question is never "does the file carry the head" but "what
do draft KV + buffers cost against what the batch knobs leave for compute" —
a per-shape answer. Measure the arm, never the flag; the booted arm table is
on [[frank-zealot]].
- **A quant's scale overhead belongs in its byte rate; doubling a measured
figure to "get the f16 number" invents 6.25 %.** q8_0 is 1.0625 B/element
(32 codes + one 2-byte scale per 32-block), f16 is exactly 2.0 ⇒ the ratio is
**1.882×**, not 2×. Derive byte rates from element count × block layout, then
check them against the boot ledger — this repo carried an invented 2× number
for weeks. Current KV math: [[frank-zealot]].
- **`--ubatch-size` has an OPTIMUM, not a plateau, and the curve is
BACKEND-SPECIFIC — never carry it across hosts.** Its compute buffers land
about TWICE as heavy on the card that holds the output layers and the drafter,
so a split balanced at one batch is unbalanced at another: re-measure the
balance after any batch change. The logical `--batch-size` is nearly free.
A large-batch allocation can fail on BUFFER CONTIGUITY while totals still fit
— no totals model can see that. Ladders: [[frank-medium-llamacpp-dflash2]]
(R9700 split) and [[frank-zealot]] (CUDA 3090).
- **On the CUDA 3090 that same knob has no interior optimum up to 2048.** A
prompt shorter than the ubatch cannot see the knob at all — that is the
control that proves the knob, not the traffic, moved the number. When a bench
IS ordered, A/B it by restarting the SERVICE through a transient
`FRANK_ZEALOT_ENV_FILE` drop-in, never a scratch port: one card holds one
server. Numbers: [[frank-zealot]].
- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same
size** (a DeltaNet boundary and a full-attention boundary can differ by ~1 GiB
each with its own KV), so balance by measurement. An affine fit reproduces the
deployed point and overstates the gaps in between.
[[frank-medium-llamacpp-dflash2]].
- **ngram-mod prints no startup banner.** The load log shows the drafter's
`adding speculative implementation 'draft-dflash'` and nothing at all for the
matcher, so a chain can look unengaged while it is working. Identify it
behaviourally (`draft_n` > 0 only where text repeats) or by the per-task
`draft acceptance = ... mean len = ...` line. It costs no VRAM (a 64 MiB
host-RAM hash table) and its window decides which traffic it can see: too
short and it proposes on ordinary prose fragments, which costs verification
without acceptance.
- **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the
per-slot number is the contract — not `ctx-size`.** `n_ctx_slot =
CTX_SIZE / PARALLEL`, and llama.cpp refuses a prompt longer than the SLOT with
a 400 even when the pool has room, so `models.yml` `contextWindow` must follow
every reslice. Both slot directions cost or buy window, never pool; shorter
slots decode slightly faster (less attention to scan, one fewer recurrent
state). Measured reslices: [[frank-zealot]].
- **Price a VRAM budget from two boots, never from a formula — and an mmproj
costs ~1.5× its file size** (the encoder's activation and compute buffers ride
with it). Subtracting two booted rows isolates it exactly; extrapolating
weights+KV+"a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7.
Then solve `KV ≤ card − non-KV − headroom` and round to a multiple of
`PARALLEL × 4096`. The two-boot table: [[frank-zealot]].
- **`images` and `image_url` are NOT interchangeable: only the content part
reaches the model.** `--mmproj` loads, `usage.prompt_tokens` rises, and the
model still invents an answer if the client used the legacy top-level
`images` field on `/v1/chat/completions` — frank-zealot answered "Brown." for
a solid-red PNG that way and "Red." through the OpenAI `image_url` part. The
failure is silent because a hallucinated color is a plausible answer. Probe
vision with a solid color and expect an exact word.
- **`--jinja` is what `tools` needs.** On 2026-09-16 a subagent died with
`500 tools param requires --jinja flag`, from the ik_llama.cpp runtime that
then served frank-dragoon: every tool-using agent on that endpoint was broken
and the failure read like a model fault. Mainline llama.cpp 0.4.1 defaults
`--jinja` on and the serve script pins it; halogen also accepts `tools` and
returns `tool_calls`. Check tool support before routing a role at an endpoint.
- **Quantized KV on CUDA needs `GGML_CUDA_FA_ALL_QUANTS=ON`** — the same
option and the same ~12x prefill collapse described in the ROCm note above.
f16 KV needs nothing; frank-zealot ran f16 for that reason until
2026-09-17, when `build-cuda` was rebuilt with the option ON so the
endpoint could take q8_0 KV.
- **llama-bench separates a LIST with `/` and SWEEP values with `,`.** Both
`-dev` and `-ts` follow this: `-dev CUDA0,Vulkan1` is two *single-device*
configurations, never one two-device run, and `-ts 0.85,0.15` is two sweep
values each becoming `[0.85, 0, …]` / `[0.15, 0, …]`, i.e. device 0 takes
everything. Write `-dev A/B -ts x/y`. This artifact produced a wrong fleet
conclusion on 2026-09-16 — "CUDA + Vulkan cannot serve one model", with two
bogus OOMs that were really "all 111 GiB on the 24 GiB card" — and the bench
table's own `dev` column was printing the truth the whole time. Any
placement-dependent number needs an independent check: sample the other
- **MTP is a property of the FILE, not of the flags.** The `-mtp` export carries the head as
`blk.<n>.nextn.*` (`qwen35.nextn_predict_layers = 1`); `--spec-type draft-mtp` sets
`load_mtp` and the draft context runs on the MAIN model's weights (`shares_model`), so there
is no second copy of a 12 GiB model in VRAM and no `-md` argument. The plain build has no MTP
tensors at all, and pointing `--model-draft` at that file loads it twice. [[frank-zealot]]
- **What can be chained: ONE context-backed drafter plus any number of draftless ngram types.**
`--spec-type a,b` chains in a fixed internal order (CLI order irrelevant) and the gate is
`ctx_dft`: the ngram family is added unconditionally while `draft-mtp`, `draft-dflash`,
`draft-dspark` and `draft-eagle3` all need a draft CONTEXT. So `draft-dflash,ngram-mod` is
legal (measured 2.6x the drafter alone on prompt-echo traffic) but `draft-mtp,draft-dflash`
dies at load — *"context type MTP requested but model doesn't contain MTP layers"*. **A LEGAL
chain can still lose on a recurrent model:** `draft-mtp,ngram-mod` on frank-dragoon's qwen35
hybrid measured −15 % prose / −7 % block-copy, because adding the ngram layer logs
`recurrent state rollback is not compatible with 'ngram-mod' - disabling rollback support` and
MTP verification depends on that rollback. Measure chains per architecture, never by analogy.
[[frank-dragoon]]
- **A drafter's KV scales with the TARGET's context, and its window policy decides the price.**
The draft context is built at `llama_n_ctx(ctx_tgt)` with no flag to cap it, so a drafter
declaring NO sliding window caches the whole window on `--spec-draft-device` and can OOM the
first draft graph capture AFTER `/health` has answered. Read `sliding_window` out of the
drafter's GGUF header before sizing, and prefer a skewed `-ts` over a context cut.
[[Llama Cpp Multi-Device Splits]]
- **A draft context bills its own f16 KV over the WHOLE POOL, not the slot**, plus its own
compute buffers, so on a single-card endpoint that squeeze can cost more than the drafter
wins. The MTP question is never "does the file carry the head" but what draft KV + buffers
cost against what the batch knobs leave for compute. Measure the arm, never the flag.
[[frank-zealot]]
- **A quant's scale overhead belongs in its byte rate; doubling a measured figure to "get the
f16 number" invents 6.25 %.** q8_0 is 1.0625 B/element (32 codes + one 2-byte scale per
32-block), f16 is exactly 2.0 ⇒ the ratio is **1.882×**, not 2×. Derive byte rates from element
count × block layout, then check them against the boot ledger — this repo carried an invented
2× number for weeks. Current KV math: [[frank-zealot]]
- **`--ubatch-size` has an OPTIMUM, not a plateau, and the curve is BACKEND-SPECIFIC — never
carry it across hosts.** Its compute buffers land about TWICE as heavy on the card that holds
the output layers and the drafter, so a split balanced at one batch is unbalanced at another:
re-measure the balance after any batch change. The logical `--batch-size` is nearly free; a
large-batch allocation can fail on BUFFER CONTIGUITY while totals still fit, which no totals
model can see. A prompt shorter than the ubatch cannot see the knob at all — that is the
control that proves the knob, not the traffic, moved the number. The 2× R9700 split peaks at
512; the CUDA 3090 has no interior optimum up to 2048. When a bench IS ordered, A/B it by
restarting the SERVICE through a transient `FRANK_<SVC>_ENV_FILE` drop-in, never a scratch
port: one card holds one server. Ladders: [[frank-medium-llamacpp-dflash2]] and
[[frank-zealot]].
- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same size** (a DeltaNet
boundary and a full-attention boundary can differ by ~1 GiB each with its own KV), so balance
by measurement. An affine fit reproduces the deployed point and overstates the gaps in between.
[[Llama Cpp Multi-Device Splits]]
- **`ngram-mod` prints no startup banner.** The load log names `draft-dflash` and nothing for the
matcher, so a chain can look unengaged while it works. Identify it behaviourally (`draft_n` > 0
only where text repeats) or by the per-task `draft acceptance = ... mean len = ...` line. It
costs no VRAM (a 64 MiB host-RAM hash table) and its window decides which traffic it can see:
too short and it proposes on ordinary prose fragments, costing verification without acceptance.
[[Llama Cpp Multi-Device Splits]]
- **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the per-slot number is
the contract — not `ctx-size`.** `n_ctx_slot = CTX_SIZE / PARALLEL`, and llama.cpp refuses a
prompt longer than the SLOT with a 400 even when the pool has room, so `models.yml`
`contextWindow` MUST follow every reslice. Both slot directions cost or buy window, never
pool; shorter slots decode slightly faster. Measured reslices: [[frank-zealot]].
- **Price a VRAM budget from two boots, never from a formula — and an mmproj costs ~1.5× its file
size** (the encoder's activation and compute buffers ride with it). Subtracting two booted rows
isolates it exactly; extrapolating weights+KV+"a bit for buffers" is what leaves 1.1 GiB free
when you meant 2.7. Then solve `KV ≤ card − non-KV − headroom` and round to a multiple of
`PARALLEL × 4096`. **The projector is NOT billed in llama.cpp's own fit check**: the engine
prints its worst case and fits WITHOUT it, so price a shape from settled `nvidia-smi` after a
real boot — the projector, and any drafter context, lands after `/health` would already have
answered (same class as the DSpark drafter OOM). Two-boot table: [[frank-zealot]].
- **`images` and `image_url` are NOT interchangeable: only the content part reaches the model.**
`--mmproj` loads, `usage.prompt_tokens` rises, and the model still invents an answer if the
client used the legacy top-level `images` field on `/v1/chat/completions` — frank-zealot
answered "Brown." for a solid-red PNG that way and "Red." through the OpenAI `image_url` part.
The failure is silent because a hallucinated color is a plausible answer; probe with a solid
color and expect an exact word. [[frank-zealot]]
- **`--jinja` is what `tools` needs.** A subagent died with `500 tools param requires --jinja
flag` on the ik_llama.cpp runtime that served frank-dragoon: every tool-using agent on that
endpoint was broken and the failure read like a model fault. Mainline llama.cpp defaults
`--jinja` on and the serve scripts pin it; halogen also accepts `tools` and returns
`tool_calls`. Check tool support before routing a role at an endpoint.
- **A model's own template can REJECT the harness's thinking level.** Qwen3.8's template accepts
only `xhigh`, `medium` and `low` for `reasoning_effort` and raises on anything else — HTTP 500,
not a clamp — so the harness's levels must be folded onto that set in `models.yml`
(`compat.reasoningEffortMap`). A role whose level is unmapped fails on its FIRST request and
the failure names Jinja, not the model: every subagent on `frank-zealot` died with
`Unexpected reasoning effort high` until `high: xhigh` was added. Halogen accepts the wider
set, so the two entries legitimately differ.
- **Thinking tokens count against `max_tokens`, so a short-budget vision probe reads as
blindness.** A probe at `max_tokens: 64` returned `content: ""` with `finish_reason: "length"`
and half a thought in `reasoning_content` while the image HAD reached the model (322 prompt
tokens vs 42 for the text alone). Budget ≥512 for any thinking model you probe for vision.
- **The FIRST request after a boot is a warmup artifact**, and the 3× spread on one shape makes
graph-capture cost read as a model fault. Benchmark the second request or later.
[[frank-zealot]]
- **A fork build hides its own dtype laws.** Vendor builds carry their own compile options, so
prove a build's KV behaviour from the KV buffer line in the boot ledger and its
`CMakeCache.txt` — never from the build flags you remember passing. [[frank-zealot]]
- **A llama.cpp endpoint is invisible to Prometheus unless `--metrics` is passed**: without it
the engine answers **501 Not Implemented** on `/metrics`, so the scrape fails while `/health`
answers 200 and `LlmEndpointDown` fires beside a healthy service — that is how `frank-zealot`
sat red across four payload generations. halogen serves `/metrics` natively, so the asymmetry
reads as "only the 3090 is broken". Diagnose with `lastError` from `/api/v1/targets`, not the
alert list, and note Alertmanager has no persistence: a reboot re-stamps `startsAt` on an alert
that has fired for weeks. [[Observability]]
- **A leftover bench drop-in silently hijacks the next deploy, and no health gate sees it.** The
sanctioned bench-window method (a transient `Environment=FRANK_<SVC>_ENV_FILE=/tmp/<arm>.env`
drop-in sourcing the repo env) outlives the bench unless retired: a leftover
`frank-zealot.service.d/ptq1.conf` made the cutover deploy boot a **different model at a
different shape** while `/health` answered 200, `/v1/models` echoed the alias (a flag, not a
fact) and the Playwright gate passed 33/33. After deploying any recently-benched LLM service:
`systemctl show <svc> -p DropInPaths` (must be empty), then
`tr '\0' ' ' </proc/$(pidof llama-server)/cmdline`, then the boot ledger — and retire drop-ins
to `~/.AGENTTRASH/<date>-<topic>/` + `daemon-reload`. [[frank-zealot]]
- **A bench verdict that never reaches the repo is indistinguishable from a rogue arm.** That
same arm served production for 24 minutes with real numbers (~48 t/s decode, 801 t/s prefill at
141k) that lived only in `/tmp` env files and journald, so the next agent read the surviving
drop-in as an unfinished experiment and reported a hijack. Write the measurement into the
endpoint page and the arm script's header in the session that takes it — `/tmp` and the journal
are not records.
- **`llama-bench` separates a LIST with `/` and SWEEP values with `,`** — for both `-dev` and
`-ts`. `-dev CUDA0,Vulkan1` is two *single-device* configurations, never one two-device run,
and `-ts 0.85,0.15` is two sweep values each putting everything on device 0. Write
`-dev A/B -ts x/y`. This artifact produced a wrong fleet conclusion ("CUDA + Vulkan cannot serve
one model") from two bogus OOMs, and the bench table's own `dev` column was printing the truth
the whole time. Any placement-dependent number needs an independent check: sample the other
device's memory (`nvidia-smi`) while the run is loaded.
- **A model's own template can REJECT the harness's thinking level.** Qwen3.8's
template accepts only `xhigh`, `medium` and `low` for `reasoning_effort` and
raises on anything else — HTTP 500, not a clamp — so the harness's levels
must be folded onto that set in `models.yml` (`compat.reasoningEffortMap`).
A role whose level is unmapped fails on its FIRST request, and the failure
names Jinja, not the model: every subagent on `frank-zealot` died with
`Unexpected reasoning effort high` until `high: xhigh` was added on
2026-09-16. Halogen accepts the wider set, so the two entries legitimately
differ.
- **The vision projector is NOT billed in llama.cpp's own fit check.** The
engine prints `[mtmd] estimated worst-case memory usage of mmproj is
873.10 MiB` and then fits WITHOUT it: frank-zealot's Bonsai-2 boot projected
21,439 MiB and settled at 22,664 (2026-10-17). Price a 3090-class shape from
settled `nvidia-smi` after a real boot, never from
`model + context + compute` — the projector (and any drafter context) lands
after `/health` would already have answered. Same failure class as the
DSpark drafter OOM.
- **The FIRST request after a boot is a warmup artifact.** Measured on
Bonsai-2/3090: 2.6 t/s prefill and 37.7 t/s decode on request #1, then
1,258 t/s prefill and 54.7 t/s (per stream, two concurrent) on the next — a
3× spread on the same shape. Benchmark the second request or later, or the
graph-capture cost reads as a model fault.
- **Thinking tokens count against `max_tokens`, so a short-budget vision probe
reads as blindness.** The Bonsai-2/`image_url` probe at `max_tokens: 64`
returned `content: ""` with `finish_reason: "length"` and half a thought in
`reasoning_content` — the image HAD reached the model (322 prompt tokens vs
42 for the text alone). At 512 tokens the same image answers "Red". Budget
≥512 for any thinking model you probe for vision.
- **Quantized KV is not the only dtype law a fork build can hide.** Vendor
builds carry their own compile options: frank-zealot's PrismML prebuilt
served `--cache-type-v q8_0` under flash attention at the exact 34,816
B/token (the `GGML_CUDA_FA_ALL_QUANTS` door), so a build decision must be
proven by the KV buffer line in the boot ledger, not by the build flags you
remember passing.
- **A llama.cpp endpoint is invisible to Prometheus unless `--metrics` is passed.**
Without it the engine answers **501 Not Implemented** on `/metrics`, so the scrape
fails while `/health` answers 200 and `LlmEndpointDown` fires beside a healthy
service — that is how `frank-zealot` sat red across four payload generations
(the oldest retained sample of `up{job="frank-zealot"}`, 2026-10-11, is already
0). halogen serves `/metrics` natively, so the asymmetry reads as "only the 3090
is broken". Diagnose with `lastError` from `/api/v1/targets`, not the alert list,
and note Alertmanager has no persistence: a reboot re-stamps `startsAt` on an
alert that has been firing for weeks.
- **A leftover bench drop-in silently hijacks the next deploy, and no health
gate sees it.** The sanctioned bench-window method — a transient
`Environment=FRANK_<SVC>_ENV_FILE=/tmp/<arm>.env` drop-in, the arm env
`source`ing the repo env and then overriding it — outlives the bench unless
it is retired. On 2026-10-18 a leftover `frank-zealot.service.d/ptq1.conf` —
the arm Frank had ordered at 02:36 and measured from 03:14 — made the cutover
deploy boot a **different model at a different shape** (3 slots of 133,888,
not 3 × 262,144) while `/health` answered 200, `/v1/models` echoed the alias
(a flag, not a fact) and the Playwright gate passed 33/33. After deploying any
recently-benched LLM service: `systemctl show <svc> -p DropInPaths` (must be
empty), then `tr '\0' ' ' </proc/$(pidof llama-server)/cmdline`, then the boot
ledger — and retire drop-ins to `~/.AGENTTRASH/<date>-<topic>/` +
`daemon-reload`.
- **A bench verdict that never reaches the repo is indistinguishable from a
rogue arm.** Same window: PTQ1_0 served production for 24 minutes with real
numbers (~48 t/s decode, 801 t/s prefill at 141k), but the figures lived only
in `/tmp` env files and journald, so the next agent on the host read the
surviving drop-in as an unfinished experiment and reported a hijack. Write the
measurement into the runbook and the arm script's header in the session that
takes it — `/tmp` and the journal are not records.
**Ops**
- Never `pkill -f` a pattern that appears in your own command line — it kills
your ssh session. Use `pkill -x <exact-name>`.
- Both hosts allow passwordless sudo for the deploy user. Use plain `sudo` in
committed scripts (no `echo pw | sudo -S`); `FRANK_SUDO_PW` covers hosts
without it.
- ComfyUI on Dankclaw runs with **GPU access** (`/dev/kfd` + `/dev/dri` via
`AddDevice`, `GroupAdd=keep-groups`, plus the udev rule that makes those nodes
world-writable — user-unit spawns carry no supplementary groups). It must NOT
pass `--cpu`: the gfx1151 ROCm wheels exist and the unit's `--cpu` era ended
2026-09-16, when a stale device-less container was found restarted 9,146
times (`RuntimeError: No HIP GPUs are available`).
- SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its
ports off the gluetun service.
- Dankclaw boots **`iommu=pt`** since 2026-10-18. The boot line is tracked in
`dankclaw/host/grub` and moves ONLY through `scripts/set-dankclaw-bootline.sh`
(diff, install, `update-grub`, verify `grub.cfg`, `--reboot`) — never by
hand-editing `/etc/default/grub`. It replaced `amd_iommu=off` (2026-09-11),
adopted on the vendor's 13–16 % prefill figure and never measured here: the
local A/B on byte-identical traffic says identity mode costs **nothing**
(+2.0 %, 1,616 t/s cold prefill), so the XDNA2 NPU returned for free —
`amdxdna` binds `c6:00.1` (PASID width 10, identity group 26) and the old
`err=-19` only ever asked for an enabled IOMMU. **AMD PerfOpt** (iommu tree
2026-09-24, Linux 7.4) still skips the page walk outright for the privileged
iGPU, but it arms only in the identity domain and 26.04 caps at `7.0.0-38`.
- Rootless Quadlet containers run with `--replace --rm`: `podman logs` covers
only the CURRENT instance; history across recreations lives in
`journalctl --user -u <name>.service`. Read rootless storage WITHOUT sudo
(per-user storage — `sudo podman logs` finds nothing).
- **A dead rootless podman pause process breaks ALL container management on the
host, and `podman system migrate` panics in 5.7.0.** Symptoms: every `podman`
command fails with `invalid internal status … could not find any running
process`, every container start fails binding podman's bridge addresses
(`Error starting server failed to bind udp listener on 10.89.x.1:53: Cannot
assign requested address`), and every `Restart=always` quadlet loops. Container
storage is intact; what is gone is `/run/user/<uid>/libpod/tmp/alive` (a UUID
token, not a pid) and the rootless netns holding the bridges — deleting the
stale token is not enough, and `podman system migrate` segfaults in
`teardownNetworkBackend`. Repair: **reboot** (`/run/user` clears and all 28
quadlets return on their own — check `Linger=yes` and `WantedBy=default.target`
first). Seen 2026-09-16, hours into heavy GPU work on the same host — but the
trigger is NOT established (`journalctl -k` recorded no OOM kill; the kernel
had no other word on it). Treat the pause process as host state that can die
for its own reasons, and reach for the reboot when it does.
- **A container cannot reach a `Network=host` service at the host's LAN IP.**
netavark DNATs *published* ports only: from a bridge container
`192.168.0.143:<published>` answers while `192.168.0.143:<host bind>` and the
gateway `10.89.1.1` are both refused. Use `host.containers.internal`
(169.254.1.2). This is what 502'd `hermes.franksdank.cloud` until the Caddyfile
upstream was changed on 2026-09-16 (its vhost comment had claimed the LAN IP
was open from that bridge — it never was).
- **A `*.gguf` filter misses mradermacher's payloads: large quants ship as
`.partNofM` fragments.** `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF`
carries Q4_K_M (37.00 + 37.00 + 36.97 = 110.97 GiB) and Q4_K_S (35.00 + 35.00
+ 34.09 = 104.09 GiB) as parts, while the smaller rungs (Q2_K, Q3_K_*, IQ3_*,
IQ4_XS) are single files. Fetch the parts, verify each against the Hub's LFS
sha256, concatenate in numeric order — that is how the deployed 111 GiB file
was assembled. Listing the Hub tree and filtering `endswith(".gguf")` makes
Q4_K_S look like it does not exist.
- **A fetch script must `-f` and verify, or it reports success on an access
error.** `curl -o FILE URL` exits 0 on HTTP 401/403 and writes the error body
as the "artifact": a gated repo yielded three 153-byte shards with identical
sha256 values and a cheerful `DOWNLOAD_DONE` in 20 seconds for 104 GiB. Use
`-f`/`--fail-with-body`, compare each file's byte count to the Hub's declared
size, and check the LFS sha256 — `scripts/download-frank-medium-gguf.sh`
already does all three; ad-hoc fetches tend not to. Gated repos (e.g.
`orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF`, same 104.09 GiB Q4_K_S)
return 403 even with the host's HF token until that account accepts the gate.
- **A scout verdict is a lead, never a finding.** In the 2026-10-19 vault de-dup
campaign, four read-only scouts worked the same pages: one classified era pages as
stale *because* they document retired payloads (that is an era page's content), one
reported files that do not exist, one reported duplications that grep disproved
(`457.9 t/s` and the ping-RTT table are solo-held by `Ethernet Bandwidth Math`),
and one cited a line number past the end of the file it judged. Re-verify every
claimed duplicate and every claimed staleness against the file and today's fleet
state before editing; a scout yield buys no evidence.
- Never `pkill -f` a pattern that appears in your own command line — it kills your ssh session.
Use `pkill -x <exact-name>`.
- Both hosts allow passwordless sudo for the deploy user. Use plain `sudo` in committed scripts
(no `echo pw | sudo -S`); `FRANK_SUDO_PW` covers hosts without it.
- **ComfyUI on Dankclaw runs with GPU access** (`/dev/kfd` + `/dev/dri` via `AddDevice`,
`GroupAdd=keep-groups`, plus the udev rule that makes those nodes world-writable — user-unit
spawns carry no supplementary groups) and must NOT pass `--cpu`: the gfx1151 ROCm wheels exist
and a stale device-less container was found restarted 9,146 times. [[ComfyUI]]
- SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its ports off the gluetun
service. [[SearXNG]] [[Gluetun]]
- **Dankclaw's kernel command line is a repo file.** It boots **`iommu=pt`** since 2026-10-18,
tracked in `dankclaw/host/grub`, and moves ONLY through `scripts/set-dankclaw-bootline.sh`
(diff, install, `update-grub`, verify `grub.cfg`, `--reboot`) — never by hand-editing
`/etc/default/grub`. It replaced `amd_iommu=off`, and the local A/B on byte-identical traffic
says identity mode costs nothing, so the XDNA2 NPU returned for free.
[[dankclaw-iommu-translation-off]]
- **Rootless Quadlet containers run with `--replace --rm`**: `podman logs` covers only the CURRENT
instance; history across recreations lives in `journalctl --user -u <name>.service`. Read
rootless storage WITHOUT sudo — it is per-user storage, so `sudo podman logs` finds nothing.
[[Quadlet Lifecycle]]
- **A dead rootless podman pause process breaks ALL container management on the host, and
`podman system migrate` panics in 5.7.0.** Every `podman` command fails with
`invalid internal status`, every start fails binding the bridges
(`failed to bind udp listener on 10.89.x.1:53`), every `Restart=always` quadlet loops — while
container storage stays intact. What is gone is `/run/user/<uid>/libpod/tmp/alive` (a UUID
token) and the rootless netns; deleting the stale token is not enough. Repair: **reboot**
(`/run/user` clears and the quadlets return on their own — check `Linger=yes` and
`WantedBy=default.target` first). The 2026-09-16 occurrence had no OOM in `journalctl -k`:
treat the pause process as host state that can die for its own reasons. [[Quadlet Lifecycle]]
- **A container cannot reach a `Network=host` service at the host's LAN IP**: netavark DNATs
*published* ports only, so the host bind and the `10.89.1.1` gateway are both refused from a
bridge container while the published port answers. Use `host.containers.internal`
(169.254.1.2). This 502'd `hermes.franksdank.cloud` until its Caddyfile upstream changed.
[[Container Host Reachability]]
- **A `*.gguf` filter misses mradermacher's payloads: large quants ship as `.partNofM`
fragments** while the smaller rungs are single files, so filtering `endswith(".gguf")` makes a
whole quant class look absent. Fetch the parts, verify each against the Hub's LFS sha256,
concatenate in numeric order. [[Model Swap and Rollback]]
- **A fetch script must `-f` and verify, or it reports success on an access error.**
`curl -o FILE URL` exits 0 on HTTP 401/403 and writes the error body as the "artifact": a gated
repo yielded three 153-byte shards with identical sha256 and a cheerful `DOWNLOAD_DONE` in 20
seconds for 104 GiB. Use `-f`/`--fail-with-body`, compare each file's byte count to the Hub's
declared size, and check the LFS sha256 (`scripts/download-frank-medium-gguf.sh` does all three;
ad-hoc fetches tend not to). Gated repos 403 even with the host's HF token until that account
accepts the gate. [[Model Swap and Rollback]]
- **A scout verdict is a lead, never a finding.** In the 2026-10-19 vault de-dup campaign six
read-only scouts worked the same pages: one classified era pages as stale *because* they
document retired payloads (that is an era page's content), one reported files that do not exist,
one reported duplications that grep disproved (`457.9 t/s` and the ping-RTT table are solo-held
by `Ethernet Bandwidth Math`), one cited a line number past the end of the file it judged, and
one drafted an Eras table with invented windows, payloads and rollback paths. Re-verify every
claimed duplicate, every claimed staleness and every quoted holder against the file and today's
fleet state before editing; a scout yield buys no evidence.
## Runbook Index
scripts/check-okf.py
Executable
+175
View File
@@ -0,0 +1,175 @@
#!/usr/bin/env python3
"""OKF v0.1 conformance linter for the docs vault (serverAppConfigs-docs).
The vault is an Open Knowledge Format bundle (omp-docs/references/okf-spec.md)
plus the maintenance discipline in okf-maintenance.md, plus this repo's own
index law (AGENTS.md -> *Documentation Policy*): an index is a locator, one row,
one clause, no numbers. Enforced here so a violation cannot be merged:
FRONTMATTER every non-reserved page has a parseable frontmatter block with a
non-empty, non-generic `type` (the spec's only hard rule)
DESCRIPTION every concept has a description that is not the title, is not a
truncation artifact (ends in an ellipsis), carries no digits and
fits the one-clause budget (<= 120 chars)
TIMESTAMP every concept has an ISO `timestamp`
INDEX-FM an index.md carries no frontmatter (it is a listing, not a concept)
INDEX-COVER every index.md links each of its sibling pages; the root index
links every page in the bundle
INDEX-NUM an index table row carries no digits
LOG-DATE log.md date headings are `## YYYY-MM-DD`, newest first
LENGTH no page exceeds 500 lines (under-split); body under 15 lines is
flagged as an over-split stub
Exit status is 1 when anything fails, so it gates a test.
"""
import datetime
import pathlib
import re
import sys
RESERVED = {"index.md", "log.md"}
GENERIC_TYPES = {"file", "document", "page", "note"}
DESC_MAX = 120
BODY_MIN = 15
LINE_MAX = 500
def split_frontmatter(text):
"""Return (frontmatter_raw_or_None, body)."""
if not text.startswith("---"):
return None, text
end = text.find("\n---", 3)
if end == -1:
return None, text
return text[3:end].strip("\n"), text[end + 4:]
def field(raw, key):
if raw is None:
return None
m = re.search(rf"^{key}:\s*(.*)$", raw, re.M)
return m.group(1).strip() if m else None
def is_iso_date(value):
v = value.strip().strip("\"'").split("+")[0].split("T")[0]
try:
datetime.date.fromisoformat(v)
return True
except ValueError:
return False
def vault_pages(vault):
return [p for p in sorted(vault.rglob("*.md"))
if ".AGENTTRASH" not in str(p) and ".obsidian" not in str(p)]
def concept_pages(vault):
return [p for p in vault_pages(vault) if p.name not in RESERVED]
def check_concepts(vault, fail):
for page in concept_pages(vault):
rel = str(page.relative_to(vault))
raw, body = split_frontmatter(page.read_text(errors="replace"))
if raw is None:
fail("FRONTMATTER", rel, "no frontmatter block")
continue
ty = field(raw, "type")
if not ty:
fail("FRONTMATTER", rel, "missing type")
elif ty.lower() in GENERIC_TYPES:
fail("FRONTMATTER", rel, f"generic type {ty!r}")
desc = (field(raw, "description") or "").strip().strip("\"'")
title = (field(raw, "title") or "").strip().strip("\"'")
if not desc or desc in ("-", "—"):
fail("DESCRIPTION", rel, "missing description")
elif desc == title:
fail("DESCRIPTION", rel, "description copies the title")
elif desc.endswith("…"):
fail("DESCRIPTION", rel, "description is a truncation artifact")
elif len(desc) > DESC_MAX:
fail("DESCRIPTION", rel, f"description is {len(desc)} chars (budget {DESC_MAX})")
elif re.search(r"\d", desc):
fail("DESCRIPTION", rel, "description carries numbers (index rows must not)")
ts = field(raw, "timestamp")
if not ts:
fail("TIMESTAMP", rel, "missing timestamp")
elif not is_iso_date(ts):
fail("TIMESTAMP", rel, f"unparseable timestamp {ts!r}")
lines = page.read_text(errors="replace").splitlines()
if len(lines) > LINE_MAX:
fail("LENGTH", rel, f"{len(lines)} lines (max {LINE_MAX})")
elif len([l for l in body.splitlines() if l.strip()]) < BODY_MIN:
fail("LENGTH", rel, "stub: fewer than %d body lines" % BODY_MIN)
def check_indexes(vault, fail):
root_index = vault / "index.md"
all_stems = {p.stem for p in concept_pages(vault)}
for index in sorted(vault.rglob("index.md")):
if ".AGENTTRASH" in str(index):
continue
rel = str(index.relative_to(vault))
text = index.read_text(errors="replace")
if text.startswith("---"):
fail("INDEX-FM", rel, "index carries frontmatter")
siblings = {p.stem for p in index.parent.glob("*.md") if p.name not in RESERVED}
for stem in sorted(siblings):
if f"[[{stem}]]" not in text:
fail("INDEX-COVER", rel, f"does not link sibling {stem}")
if index == root_index:
for stem in sorted(all_stems - {p.stem for p in index.parent.glob("*.md")}):
if f"[[{stem}]]" not in text:
fail("INDEX-COVER", rel, f"does not link {stem}")
for line in text.splitlines():
if line.startswith("|") and not re.match(r"^\|[\s-|]+$", line) and "Page" not in line:
if re.search(r"\d", line):
fail("INDEX-NUM", rel, f"row carries numbers: {line[:70]}")
def check_log(vault, fail):
for log in sorted(vault.rglob("log.md")):
if ".AGENTTRASH" in str(log):
continue
rel = str(log.relative_to(vault))
dates = []
for line in log.read_text(errors="replace").splitlines():
m = re.match(r"^##\s+(.+)$", line)
if not m:
continue
head = m.group(1).strip()
dm = re.match(r"^(\d{4}-\d{2}-\d{2})$", head)
if not dm:
fail("LOG-DATE", rel, f"heading {head!r} is not a bare YYYY-MM-DD")
continue
if not is_iso_date(dm.group(1)):
fail("LOG-DATE", rel, f"heading {head!r} is not a real date")
dates.append(dm.group(1))
if dates != sorted(dates, reverse=True):
bad = next(i for i in range(1, len(dates)) if dates[i] > dates[i - 1])
fail("LOG-DATE", rel, f"{dates[bad]} after {dates[bad - 1]}: not newest first")
def main(argv):
vault = pathlib.Path(argv[1] if len(argv) > 1 else "serverAppConfigs-docs")
if not vault.is_dir():
print(f"not a directory: {vault}")
return 2
failures = []
def fail(rule, page, msg):
failures.append(f"{rule:12} {page}: {msg}")
check_concepts(vault, fail)
check_indexes(vault, fail)
check_log(vault, fail)
print(f"{len(concept_pages(vault))} concept pages, {len(failures)} OKF violations")
for line in failures:
print(" " + line)
return 1 if failures else 0
if __name__ == "__main__":
sys.exit(main(sys.argv))
scripts/gen-vault-index.py
Executable
+135
View File
@@ -0,0 +1,135 @@
#!/usr/bin/env python3
"""Regenerate serverAppConfigs-docs/index.md from page frontmatter.
The root index is a locator (AGENTS.md -> *Documentation Policy*): one row per
page, one clause, no numbers. Hand-maintaining it is how rows go missing, and
the docs seam fails when a page is not linked from it, so it is generated:
* preamble everything above the first `## ` in the existing file is kept
verbatim (that is the curated statement of the vault's law)
* section from the page's directory, overridden by its frontmatter tags
(`endpoint` / `retired-endpoint` split the LLM pages)
* row text the page's frontmatter `description`, verbatim
A page with no description is a hard error rather than an empty cell, so the
generator cannot silently produce a silent index. Sections with no pages are
skipped. Deterministic: the same vault always yields the same bytes.
"""
import pathlib
import re
import sys
RESERVED = {"index.md", "log.md"}
SECTIONS = [
("Inference endpoints", "endpoint"),
("Retired endpoints", "retired-endpoint"),
]
DIRECTORY_SECTIONS = [
("Dankclaw", "Dankclaw services"),
("Dankserver", "Dankserver services"),
("FrankBook-Pro", "FrankBook-Pro"),
("Desktop", "Workstations"),
("Infrastructure", "Cross-host infrastructure"),
("Architecture", "Repo architecture and policy"),
("Operations", "Operations and incidents"),
("Agent", "Agent / harness"),
("Research", "Research and methodology"),
]
PIPELINE_SECTION = "Pipeline Parallelism (series)"
LLM_FALLBACK = "LLM technique and hardware references"
CHANGE_SECTION = "Change history"
ORDER = [t for t, _ in SECTIONS] + [t for _, t in DIRECTORY_SECTIONS] + \
[PIPELINE_SECTION, LLM_FALLBACK, CHANGE_SECTION]
def frontmatter_fields(text):
if not text.startswith("---"):
return {}
end = text.find("\n---", 3)
if end == -1:
return {}
raw = text[3:end]
fields = {}
for key in ("description", "type"):
m = re.search(rf"^{key}:\s*(.*)$", raw, re.M)
if m:
fields[key] = m.group(1).strip().strip("\"'")
tags = []
m = re.search(r"^tags:\s*\[(.*?)\]$", raw, re.M)
if m:
tags = [t.strip().strip("\"'") for t in m.group(1).split(",") if t.strip()]
elif re.search(r"^tags:\s*$", raw, re.M):
block = re.search(r"^tags:\s*\n((?:\s+-\s+.*\n?)+)", raw, re.M)
if block:
tags = [l.strip().lstrip("-").strip().strip("\"'")
for l in block.group(1).splitlines() if l.strip()]
fields["tags"] = tags
return fields
def section_for(page, vault, fields):
rel = page.relative_to(vault)
if len(rel.parts) == 1:
return CHANGE_SECTION if page.name == "log.md" else "Uncategorised"
top = rel.parts[0]
if top == "LLM":
if PIPELINE_SECTION_DIR in rel.parts:
return PIPELINE_SECTION
for title, tag in SECTIONS:
if tag in fields.get("tags", []):
return title
return LLM_FALLBACK
for prefix, title in DIRECTORY_SECTIONS:
if top == prefix:
return title
return "Uncategorised"
PIPELINE_SECTION_DIR = "Pipeline Parallelism"
def main(argv):
vault = pathlib.Path(argv[1] if len(argv) > 1 else "serverAppConfigs-docs")
index = vault / "index.md"
if not index.is_file():
print(f"missing {index}")
return 2
existing = index.read_text()
preamble = existing.split("\n## ", 1)[0].rstrip("\n")
pages = [p for p in vault.rglob("*.md")
if ".AGENTTRASH" not in str(p) and ".obsidian" not in str(p)
and p.name != "index.md" and p != index]
buckets, errors = {}, []
for page in pages:
fields = frontmatter_fields(page.read_text(errors="replace"))
if page.name != "log.md":
desc = fields.get("description")
if not desc:
errors.append(str(page.relative_to(vault)))
continue
else:
desc = "Chronological incident and law log for the whole bundle."
buckets.setdefault(section_for(page, vault, fields), []).append((page.stem, desc))
if errors:
print("pages with no description (fix the frontmatter, not the index):")
for e in sorted(errors):
print(" " + e)
return 1
out = [preamble, ""]
for title in ORDER + sorted(set(buckets) - set(ORDER)):
rows = sorted(buckets.get(title, []), key=lambda r: r[0].lower())
if not rows:
continue
out += [f"## {title}", "", "| Page | Owns |", "|---|---|"]
out += [f"| [[{stem}]] | {desc} |" for stem, desc in rows]
out.append("")
out.append("<!-- generated by scripts/gen-vault-index.py; edit page frontmatter instead -->")
index.write_text("\n".join(out) + "\n")
print(f"{index}: {sum(len(v) for v in buckets.values())} rows in {len([t for t in ORDER if buckets.get(t)])} sections")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))
serverAppConfigs-docs/LLM/frank-zealot.md
+10 -5
View File
@@ -98,7 +98,7 @@ re-run the same arms with the box quiet before a figure is carried anywhere else
| Window | Payload | Runtime | Shape | Replaced because | Rollback |
|---|---|---|---|---|---|
| 2026-10-18 13:19 → same evening (the boot `68afc6d` replaced) | the same 9B MTP file and projector | PrismML fork 10777 | 3 × 262,144 (pool 786,432), q8_0 K+V = 13,056 MiB, settled 17,956 / 6,169 MiB; 168.7 t/s decode (three runs 168.4–168.7), ~300 t/s over three streams (102.8 each), 2,877 t/s prefill at 106,418 | Frank ordered unquantized KV, and that pool cannot survive it: 24,576 MiB of f16 cache > 24,123 addressable | one env line — `CTX_SIZE=786432 PARALLEL=3 CACHE_TYPE_K/V=q8_0` — and redeploy; no payload move |
| 2026-10-18, 24 min (arm under the unit) | `Bonsai2-PTQ1_0/Ternary-Bonsai-2-27B-PTQ1_0.gguf` 5,946,648,928 B + hardlinked Q8 mmproj | PrismML fork 10777 | 2 × 200,704, batch/ubatch 3072/1536 | Frank scratched the 27B plan — ~48 t/s decode vs 76.6 for PQ2_0 on the same weights | `/home/dank/models/Bonsai2-PTQ1_0/` + `scripts/download-frank-zealot-bonsai2-ptq1.sh` |
| 2026-10-18 03:14 → 13:19 (arm under the unit, then the leftover drop-in) | `Bonsai2-PTQ1_0/Ternary-Bonsai-2-27B-PTQ1_0.gguf` 5,946,648,928 B + hardlinked Q8 mmproj | PrismML fork 10777 | arm: 2 × 200,704, batch/ubatch 3072/1536. The leftover `frank-zealot.service.d/ptq1.conf` then made the 13:16 cutover deploy boot THIS payload at 3 × 133,888 (`new slot, n_ctx = 133888` x3, journal read 2026-10-19) until the 13:19 boot | Frank scratched the 27B plan — ~48 t/s decode vs 76.6 for PQ2_0 on the same weights | `/home/dank/models/Bonsai2-PTQ1_0/` + `scripts/download-frank-zealot-bonsai2-ptq1.sh` |
| 2026-10-17 → 10-18 | Ternary-Bonsai-2-**27B** PQ2_0 7,206,168,928 B `3907dc16…` + Q8_0 mmproj 629,246,976 B | PrismML fork **10754** (`~/GitPackages/llama-prism/`) | 2 × 200,704 (pool 401,408), q8_0 K+V = 13,328 MiB, settled 22,664 / 1,461; 76.6 t/s decode, 109.4 aggregate over two streams, 691.7 t/s prefill at 166,027; MTP impossible — no head in the file | the 9B bills half the KV per token (8 full layers, not 16), which funded three slots at the native window and beat it on every rate | `/home/dank/models/Bonsai2-PQ2_0/` + the 10754 fork tree, `scripts/download-frank-zealot-bonsai2.sh`, revert the swap commit |
| 2026-10-07 → 10-17 | Qwen3.5-9B-Uncensored-HauhauCS-Aggressive Q6_K 7,359,259,008 B `c0ba7beb…` + BF16 mmproj 921,704,448 B | llama.cpp master `5fc4f3c` CUDA sm_86 (`~/GitPackages/llama-master/build-cuda/`) | 2 × 200,704, exact f16 K+V, MTP off (no `nextn` tensors) | the ternary 27B bought a larger model class at the same window | `/home/dank/models/Qwen35-9B-HauhauCS-Q6K/` + the master build, `scripts/download-frank-zealot.sh` |
| 2026-10-07 (same day) | Gemma4-26B-A4B-QAT Q4_K_M 16,796,015,520 B + BF16 mmproj 1,194,827,776 B + unused `gemma4-assistant` MTP 251,937,728 B | llama.cpp master CUDA | 1 × 200,704, f16 K + q8_0 V, ubatch 1024, MTP off; 21,868 / 2,257 settled, 139.9 t/s decode (the ~4 B-active MoE is at home here); MTP ON would cost ~1.9 GiB and boot at 319 MiB free | ONE slot serialized compaction; the 9B's lighter weights bought two slots and exact KV | `/home/dank/models/Gemma4-26B-A4B/` (ungated — plain `curl -fL`) |
@@ -129,10 +129,12 @@ re-run the same arms with the box quiet before a figure is carried anywhere else
3 cells, and an armed depth-2 drafter triples it to 452.25), compute buffers at graph capture, the mmproj after the
fit. On gemma4-class hybrids `attention_k_eq_v` does **not** shrink the allocation — the cache reserves the K/V pair
twice, 6.125 B/elem at f16 K + q8_0 V.
- **`--flash-attn on` is pinned, not decorative**: a quantized V exists only on the FA path and the engine falls back silently without it. llama.cpp has **no true fp8 KV dtype** — q8_0 is the 8-bit class (1.0625 B/elem, ~8.5 effective bits). `GGML_CUDA_FA_ALL_QUANTS` is vestigial and build-specific: the PrismML fork carries it ON (since 2026-09-17) while the master build had it OFF and still dispatched Q4/Q5 KV unconditionally (`ggml-cuda/fattn.cu` has no guard), so quantized KV never required it.
- **`--flash-attn on` is pinned, not decorative**: a quantized V exists only on the FA path and the engine falls back silently without it. llama.cpp has **no true fp8 KV dtype** — q8_0 is the 8-bit class (1.0625 B/elem, ~8.5 effective bits). `GGML_CUDA_FA_ALL_QUANTS` is **deprecated upstream and inert in both trees on this host** (no reference anywhere in `ggml/src/ggml-cuda/`, read 2026-10-19): master's `ggml/cmake/common.cmake` warns `use GGML_CUDA_FA_QUANTS=all instead`, and the successor's default — `GGML_CUDA_FA_QUANTS="q4_0-q4_0;q8_0-q8_0;f16-f16;bf16-bf16"` (`ggml/CMakeLists.txt:208`) — already compiles the quantized-KV kernels with the old flag OFF (master's `build-cuda` cache says `FA_ALL_QUANTS:BOOL=OFF` and it dispatched q8_0/q4_0 KV regardless). Prove a build's quantized-KV ability from its `CMakeCache.txt` `GGML_CUDA_FA_QUANTS` line, never from the retired flag or the build command you remember.
- **Prices come from boots, not formulas, and an arm belongs to the payload it was measured on**: 3 × 180,224 +
q8_0 V fits with Q5_K_M (21,684 used / 2,441 free) and does **not** fit with Q6_K inside the 2–3 GiB band. The
mmproj worst case (~870 MiB) is printed before fitting but never billed by the engine's own fit check. Only
q8_0 V fits with Q5_K_M (21,684 used / 2,441 free) and does **not** fit with Q6_K inside the 2–3 GiB band.
The mmproj worst case is printed before fitting (`estimated worst-case memory usage of mmproj is
873.10 MiB` on the Bonsai-2 27B boot) but never billed by the engine's own fit check: that boot
projected **21,439 MiB** and settled at **22,664**. Only
`tests/frank_zealot_unit.bats`'s fit gate can see a boot that would die at `cudaMalloc`, so every retune keeps it
green — and the first verbosity-4 ledger plus settled `nvidia-smi` of a new dtype belongs on this page, because
projections drift.
@@ -179,7 +181,10 @@ re-run the same arms with the box quiet before a figure is carried anywhere else
`predicted_per_second`). Keep `VERBOSITY=4` — 3 is the engine default and cannot prove the KV or draft sizes,
while levels above 4 flood the journal.
- llama.cpp does **not** validate the requested model id: it answers and echoes the served `ALIAS`, so a rename breaks id-keyed consumers (the OpenWebUI `model` row, its `access_grant`, `openai.api_configs`) and nothing server-side. Alias history: `frank-small` → `frank-secondary` (2026-10-06) → `frank-zealot` (`7c77326`, 2026-10-09); the service and unit took the name 2026-10-10 (`55f87fc`).
- The first request after a boot is a warmup artifact (37 t/s class) and is never a measurement; never time a wall-clock aggregate on a generation that stops on its own (use `ignore_eos`).
- The first request after a boot is a warmup artifact, never a measurement: on the Bonsai-2 27B
boot request #1 gave 2.6 t/s prefill / 37.7 t/s decode and request #2 gave 1,258 t/s prefill /
54.7 t/s per stream (two concurrent, the 109.4 aggregate in the Eras row) — a 3x spread on one
shape. Never time a wall-clock aggregate on a generation that stops on its own (use `ignore_eos`).
## Files