docs(agents): compress Key Lessons to one law per bullet, correct the FA-quants law
Key Lessons Learned 483 -> 340 lines. Every law survives; only the essays went. Port-before-cut census over the diff: 454 deleted lines, 0 orphaned facts (every number still has a holder in the vault). - Each lesson is now one bullet with the page that owns the detail. - De-duplicated the page's own internal repeats: the layer-split law was told twice (PCIe and ROCm sections), the FA-quants law twice more. - CORRECTION, verified on dankclaw 2026-10-19: `GGML_CUDA_FA_ALL_QUANTS` is deprecated upstream (master's ggml/cmake/common.cmake tells you to use GGML_CUDA_FA_QUANTS=all), the successor's default pair list `q4_0-q4_0;q8_0-q8_0;f16-f16;bf16-bf16` compiles the quantized-KV kernels with the old flag OFF (master build-cuda cache: FA_ALL_QUANTS=OFF), and neither llama tree references the old macro under ggml/src/ggml-cuda/. The "required or prefill collapses ~12x" law is obsolete; the new law is to read the CMakeCache pair list. - frank-zealot: ported the three facts AGENTS.md alone carried, and verified them against the host first — the 2026-10-18 13:16 cutover boot really did load PTQ1_0 at `new slot, n_ctx = 133888` x3 (journal), the mmproj fit check really printed 873.10 MiB against a 21,439 MiB projection, and the warmup spread is 2.6 -> 1,258 t/s prefill. - New tooling: scripts/check-okf.py (OKF v0.1 conformance linter) and scripts/gen-vault-index.py (root index regenerated from frontmatter). Co-Authored-By: omp <omq@openclaw.sh>
This commit is contained in:
Frank
omp
@@ -475,481 +475,338 @@ linked runbooks.
|
|||||||
|
|
||||||
**PCIe / multi-GPU**
|
**PCIe / multi-GPU**
|
||||||
|
|
||||||
- **RCCL collectives need PCIe atomics on every participating GPU.** RCCL's
|
- **RCCL collectives need PCIe atomics on every participating GPU, and the failure never names
|
||||||
kernels carry a hidden hostcall-buffer parameter; ROCr refuses to dispatch
|
atomics.** The kernels carry a hidden hostcall-buffer parameter; ROCr refuses to dispatch them on
|
||||||
them when the device has no atomics, and the failure surfaces as
|
a device without atomics and the error surfaces as `hipErrorIllegalState` from `hipLaunchKernel`,
|
||||||
`hipErrorIllegalState` ("the operation cannot be performed in the present
|
not as anything that names atomics. The value is consumed at driver PROBE time, so writing the
|
||||||
state") from `hipLaunchKernel`, not as anything that names atomics. The
|
AtomicOp Requester bit with `setpci` at runtime changes nothing. Check the CHAIN: `lspci -vv -s
|
||||||
value is consumed at driver probe time, so writing the AtomicOp Requester
|
<gpu>` and every upstream bridge must show `AtomicOpsCap: Routing+` (a `Routing-` bridge is a
|
||||||
bit with `setpci` at runtime changes nothing. Check the *chain*:
|
hardware wall), and the endpoint's Device Control 2 (`setpci -s <gpu> CAP_EXP+28.w`) must have bit
|
||||||
`lspci -vv -s <gpu>` and every upstream bridge must show
|
6 set. AMD desktop platforms put one card on CPU root ports (atomics work) and the rest behind the
|
||||||
`AtomicOpsCap: Routing+` (a `Routing-` bridge is a hardware wall), and the
|
chipset (usually not) — that asymmetry is what blocked TP=2 here. A one-card collective proves
|
||||||
endpoint's Device Control 2 (`setpci -s <gpu> CAP_EXP+28.w`) must have bit
|
nothing: NCCL short-circuits at `world_size=1`. [[PCIe Link and Atomics]]
|
||||||
6 set. AMD desktop platforms put one card on CPU root ports (atomics work)
|
- **That wall was physical, and a board swap removed it on 2026-09-20.** dankserver took the path of
|
||||||
and the rest behind the chipset (usually not) — that asymmetry is what
|
a board whose CPU lanes bifurcate to x8/x8 Gen5, both cards on root ports, every bridge `Routing+`
|
||||||
blocked frank-medium's TP=2. A one-card collective proves nothing: NCCL
|
and BOTH endpoints `ReqEn+` — the exact precondition the old test plan named. Bandwidth was never
|
||||||
short-circuits at `world_size=1`.
|
the constraint (a 27B TP=2 allreduce at batch 4 is ≈10 MB per token-step, under 0.4 ms at Gen5 x8).
|
||||||
- **That wall was physical, and it was fixed on 2026-09-20 by a board swap.**
|
The collective choice matters: TheRock's RCCL builds lack gfx1201 device code
|
||||||
Both paths that put two cards on CPU root ports were: an M.2→PCIe x4 riser
|
(`invalid device function`), which is why vLLM runs its R4D one-shot allreduce here at all.
|
||||||
into a *CPU-attached* M.2 slot, or a board whose CPU lanes bifurcate to
|
Evidence matrix and the measured three-arm verdict: [[Dual-R9700-Cooperation]].
|
||||||
x8/x8. dankserver took the second: **ASUS ProArt B850-Creator WiFi NEO**,
|
- **When a GPU collective dies, read the runtime, not the framework.** The message that identified
|
||||||
`PCIEX16_1` + `PCIEX16_2` at x8/x8 Gen5 on root ports `00:01.1` / `00:01.3`,
|
the cause came from `AMD_LOG_LEVEL=3` (`Pcie atomics not enabled, hostcall not supported` →
|
||||||
every bridge now `Routing+` and **both endpoints `ReqEn+`** — the exact
|
`AQL dispatch failed!`), never from vLLM or RCCL — the env var costs one line and prints the
|
||||||
precondition the old test plan named. Bandwidth was never the constraint (a
|
failing HIP call with its return code. [[PCIe Link and Atomics]]
|
||||||
27B TP=2 allreduce at batch 4 is ≈10 MB per token-step, ≈1.4 ms at Gen4 x4
|
- **Layer split is the no-collective path, on one vendor's cards and across two — and
|
||||||
and under 0.4 ms at Gen5 x8).
|
`--override-tensor` is what cannot ride with it.** Layers go in order and only the boundary
|
||||||
**Proven on the same day.** vLLM's R4D one-shot allreduce runs TP=2 here and
|
activation crosses PCIe once per forward pass, so it works wherever a collective cannot be
|
||||||
frank-medium serves on it; llama.cpp's RCCL path was built too
|
dispatched (and on an RCCL-less fork it beats tensor mode, which falls back to a butterfly
|
||||||
(`GGML_HIP_RCCL=ON`) and its tensor mode measured — real collective, slower
|
allreduce); it is also the cross-vendor answer, where a CUDA + Vulkan split held a real
|
||||||
shape. Note the collective choice matters: TheRock's RCCL builds lack gfx1201
|
two-device graph. But `--override-tensor <group>=<device>` sets `has_tensor_overrides()`, a
|
||||||
device code (`invalid device function`), which is why R4D is the path vLLM
|
REQUIRED term of the scheduler's `pipeline_parallel` flag (llama-context.cpp), and that flag sets
|
||||||
uses at all. Evidence matrix and the measured three-arm verdict:
|
`n_copies` in `ggml_backend_sched_new`: with `n_copies == 1` a pre-allocated weight on a
|
||||||
[[Dual-R9700-Cooperation]].
|
non-primary device has no backend and the graph aborts (ggml-backend.cpp, reported with the leaf's
|
||||||
- **When a GPU collective dies, read the runtime, not the framework.** The
|
own op name `(NONE)`). `=CPU` is the only override target that works, because the CPU backend is
|
||||||
message that identified the cause came from `AMD_LOG_LEVEL=3`
|
last in the list, and pure Vulkan changes nothing — the gate is the override, not the driver mix.
|
||||||
(`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch
|
Pin the spec-decode drafter to one device: an unpinned draft under tensor mode aborts the KV build,
|
||||||
failed!`), not from vLLM or RCCL. `AMD_LOG_LEVEL` costs one env var and
|
and on a split its home is the card holding the target's last layers.
|
||||||
prints the failing HIP call with its return code.
|
[[Llama Cpp Multi-Device Splits]] [[x4-pair-experiment]]
|
||||||
- **The software path that puts both cards on ONE model without a collective:
|
- **A host↔3090 link at x4 measures 6.72 GB/s, and that is a design constraint, not a detail.**
|
||||||
llama.cpp's layer split.** On the old board vLLM could not do it in any of its
|
Pinned H2D 6.72 / D2H 6.60 GB/s (≈85 % of the Gen4 x4 ceiling), async no better than sync and
|
||||||
parallelisms — TP needs an allreduce, PP needs the same p2p primitive, and both
|
pageable within 1 % of pinned — the copies are LINK-limited, not staging-limited, so the usual
|
||||||
died on the card that could not dispatch an RCCL kernel. `--split-mode
|
"pageable costs 2×" rule does not apply. The card and the port beneath it are both wider than x4,
|
||||||
layer` in llama.cpp needs no
|
so the x4 lives in the adapter or cable, and the link idles at 2.5 GT/s until load. Consequence
|
||||||
collective: layers are distributed in order and only the boundary activation
|
for any engine that STREAMS weights across it: a 3 MiB slab costs 474 µs, so a 111 GiB A3B model
|
||||||
crosses PCIe, once per forward pass. Measured here: 1991 t/s prefill /
|
needing ~1.70 GiB of active experts per token lands at 3.7 tok/s, against 20.46 tok/s for the same
|
||||||
18.8 tok/s decode layer-split, against 272 / 26.3 tensor (butterfly
|
model with the experts COMPUTED in host RAM (`-ot exps=CPU`, where the link carries activations
|
||||||
fallback).
|
only). Streaming over PCIe loses; computing in RAM does not care. [[x4-pair-experiment]]
|
||||||
This was what `frank-medium` ran until 2026-09-20, when the board swap made
|
|
||||||
collectives legal and radiance TP=2 beat it on every axis (95.8 vs 45.7 t/s
|
|
||||||
decode, 0.21 vs 0.72 s TTFT, 259 vs 72 t/s burst-4) — see
|
|
||||||
[[frank-medium-radiance-mxfp4]]; the llama shape is the rollback target
|
|
||||||
[[frank-medium-llamacpp-dflash2]].
|
|
||||||
- **Layer split is also the cross-vendor answer — and `-ot` is what cannot be
|
|
||||||
combined with it.** On Dankclaw (2026-09-16) a CUDA 3090 + Vulkan iGPU layer
|
|
||||||
split ran a real two-device graph: `-dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85`
|
|
||||||
held 14,994 MiB on the 3090 and gave 27.2 tok/s with 268.5 t/s prefill at 64k
|
|
||||||
depth (iGPU alone: 25.8 / 231.8). What does NOT work is per-tensor placement
|
|
||||||
across devices: `--override-tensor <group>=<device>` sets
|
|
||||||
`has_tensor_overrides()`, a REQUIRED term of the scheduler's
|
|
||||||
`pipeline_parallel` flag (llama-context.cpp:433 — the only place in the tree
|
|
||||||
that reads it), and that flag is what sets `n_copies` in
|
|
||||||
`ggml_backend_sched_new`; with `n_copies == 1` a pre-allocated weight on a
|
|
||||||
non-primary device has no backend and the graph aborts (`ggml-backend.cpp:941`,
|
|
||||||
reported with the leaf's own op name `(NONE)`). `=CPU` is the only override
|
|
||||||
target that works, because the CPU backend is last in the list. Pure Vulkan
|
|
||||||
changes nothing here — the gate is the override, not the driver mix. See
|
|
||||||
[[x4-pair-experiment]].
|
|
||||||
- **A host↔3090 link at x4 measures 6.72 GB/s, and that is a design
|
|
||||||
constraint, not a detail.** `pcie-link-test.cu` on Dankclaw (2026-09-16,
|
|
||||||
services stopped): pinned H2D 6.72 GB/s, D2H 6.60 GB/s — 85 % of the Gen4 x4
|
|
||||||
ceiling — with async no better than sync and pageable within 1 % of pinned,
|
|
||||||
i.e. the copies are link-limited rather than staging-limited (the usual
|
|
||||||
"pageable costs 2x" rule does not apply here). The card is x16-capable and the
|
|
||||||
Strix Halo port beneath it is x8-capable (`LnkCap: 16GT/s, Width x8` on
|
|
||||||
`00:03.1`), so the x4 lives in the adapter or cable; the link idles at
|
|
||||||
2.5 GT/s x4 and trains up under load. Consequence for any engine that
|
|
||||||
*streams* weights across it: a 3 MiB slab costs 474 us, and a 111 GiB A3B
|
|
||||||
model needs ~1.70 GiB of active experts per token ⇒ **3.7 tok/s** (4.6 with a
|
|
||||||
20 % resident cache), against **20.46 tok/s** for the same model with the
|
|
||||||
experts *computed* in host RAM by llama.cpp (`-ot exps=CPU`, whence the link
|
|
||||||
carries activations only) and 25–26.5 on the iGPU. Streaming over PCIe loses;
|
|
||||||
computing in RAM does not care. FreeToken — expert LRU in host RAM, exactly
|
|
||||||
this streaming shape — also loads only modelopt block-FP8 safetensors for
|
|
||||||
`qwen3_5_moe`, so a Q4_K_M GGUF is unloadable there before bandwidth is even
|
|
||||||
the question. See [[x4-pair-experiment]].
|
|
||||||
|
|
||||||
|
|
||||||
**ROCm / GPU**
|
**ROCm / GPU**
|
||||||
|
|
||||||
- **Layer split needs no collective** — layers go in order and only the boundary
|
- **Bundling a ROCm runtime next to a llama.cpp binary: `cp -r SRC DST/` is a trap.** With a missing
|
||||||
activation crosses PCIe, so it is the fallback wherever a collective cannot be
|
`DST`, cp creates `DST` as a copy of `SRC` and drops one directory level. rocBLAS and hipBLASLt
|
||||||
dispatched (and on an RCCL-less fork it beats tensor mode, which falls back to
|
look for their device code at a FIXED depth (`../lib/rocblas/library`,
|
||||||
a butterfly allreduce). Pin the spec-decode drafter to one device: an unpinned
|
`../lib/hipblaslt/library/<arch>`), so the flattened copy fails at REQUEST time, not load time:
|
||||||
draft under tensor mode aborts the KV build, and on a split its home is the
|
the engine reads the weights, answers `/health` and dies on the first completion with
|
||||||
card holding the target's last layers. Deployed history:
|
`Cannot read .../TensileLibrary.dat: No such file or directory for GPU arch : gfx1201`. Declare
|
||||||
[[frank-medium-llamacpp-dflash2]].
|
the destination, then copy the contents.
|
||||||
- **Bundling a ROCm runtime next to a llama.cpp binary: `cp -r SRC DST/` is a
|
- **mmap must stay on** for the big-PLE models: `--no-mmap` forces the ~50 GiB PLE table into
|
||||||
trap.** With a missing `DST`, cp creates `DST` as a copy of `SRC` and drops
|
anonymous memory → silent cgroup kill, with dmesg as the only evidence. mmap pages ARE page cache,
|
||||||
one directory level. rocBLAS and hipBLASLt look for their device code at a
|
and pinning the PLE tensor with `-ot` cost 23 → 13.4 t/s. [[Llama Cpp Serving Sizing]]
|
||||||
fixed depth (`../lib/rocblas/library`, `../lib/hipblaslt/library/<arch>`),
|
- **HIP reads bare env vars** `TEMP`, `HOST`, `GPU_MAX_HEAP_SIZE`. Serve scripts export env files
|
||||||
and the flattened copy fails at *request* time, not load time: the engine
|
with `set -a`, so a tunable named `TEMP` leaks into HIP (`TEMP=1.0` sent the JIT cache to path
|
||||||
reads the weights, answers `/health` and dies on the first completion with
|
`1.0` and segfaulted every load, fixed 2026-08-13 by renaming to `TEMPERATURE`). Never name a
|
||||||
`Cannot read .../TensileLibrary.dat: No such file or directory for GPU arch
|
tunable these; check with
|
||||||
: gfx1201`. Declare the destination, then copy the contents.
|
|
||||||
- **mmap must stay on** for the big-PLE models: `--no-mmap` forces the
|
|
||||||
~50 GiB PLE table into anonymous memory → silent cgroup kill, dmesg is the
|
|
||||||
only evidence. mmap pages ARE page cache. Never pin the PLE tensor with
|
|
||||||
`-ot` (23 → 13.4 t/s).
|
|
||||||
- **HIP reads bare env vars** `TEMP`, `HOST`, `GPU_MAX_HEAP_SIZE`. Serve
|
|
||||||
scripts export env files with `set -a`, so a tunable named `TEMP` leaks
|
|
||||||
into HIP (`TEMP=1.0` sent the JIT cache to path `1.0` and segfaulted every
|
|
||||||
load, fixed 2026-08-13 by renaming to `TEMPERATURE`). Never name a tunable
|
|
||||||
these; check with
|
|
||||||
`strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE"`.
|
`strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE"`.
|
||||||
- **`-DGGML_CUDA_FA_ALL_QUANTS=ON` is required** for q8_0 KV + flash
|
- **`GGML_CUDA_FA_ALL_QUANTS` is dead; do not build for it.** Verified on this host 2026-10-19:
|
||||||
attention on CUDA builds; without it prefill collapses ~12x (95 vs 1300+
|
master's `ggml/cmake/common.cmake` warns it is deprecated in favour of `GGML_CUDA_FA_QUANTS`, the
|
||||||
t/s, measured 2026-08-17).
|
successor's DEFAULT pair list (`q4_0-q4_0;q8_0-q8_0;f16-f16;bf16-bf16`) already compiles the
|
||||||
- **Flash-Next MTP heads do not load on any current loader** — they need full
|
quantized-KV kernels with the old flag OFF, and neither llama tree on this host references the old
|
||||||
per-layer `blk.<il>.hc_attn_norm.weight` and ship only `blk.48` (verified
|
macro anywhere under `ggml/src/ggml-cuda/`. The old "~12× prefill collapse" belongs to the older
|
||||||
3x). Don't re-attempt without new loader support.
|
source line. Prove a build's KV ability from its `CMakeCache.txt`. [[frank-zealot]]
|
||||||
- **QSA flash-attn assert:** `GGML_ASSERT(a-type == GGML_TYPE_F32)` on
|
- **Flash-Next MTP heads do not load on any current loader** — they need full per-layer
|
||||||
qwen4exp needs `scripts/patches/qwen4exp-qsa-f32-mask.patch` — but only on
|
`blk.<il>.hc_attn_norm.weight` and ship only `blk.48` (verified 3×). Do not re-attempt without new
|
||||||
the agentionai FAST layer set; the kingjones STRIX set works unpatched.
|
loader support.
|
||||||
- **GPU budget on Dankclaw is the GTT aperture (~114 GiB), not the
|
- **QSA flash-attn assert:** `GGML_ASSERT(a-type == GGML_TYPE_F32)` on qwen4exp needs
|
||||||
rocm-smi "dedicated" figure.** The Radeon 8060S (gfx1151) reports
|
`scripts/patches/qwen4exp-qsa-f32-mask.patch` — but only on the agentionai FAST layer set; the
|
||||||
116,732 MiB (`llama-server --list-devices`;
|
kingjones STRIX set works unpatched. [[Llama Cpp Serving Sizing]]
|
||||||
`gtt_total=122402963456` in `/sys/class/drm/card1/device/`). The older
|
- **The GPU budget on Dankclaw is the GTT aperture, not the rocm-smi "dedicated" figure.** The
|
||||||
"64 GiB, SVM-backed" lesson was already stale — the CIRU Orca predecessor
|
Radeon 8060S (gfx1151) reports 116,732 MiB (`llama-server --list-devices`; `gtt_total` in
|
||||||
held 92.6 GiB of weights on the iGPU. `free`/`dmidecode` show the
|
`/sys/class/drm/card1/device/`), and the older "64 GiB, SVM-backed" lesson was already stale.
|
||||||
CPU-side carve-out — intentional, do not change.
|
`free`/`dmidecode` show the CPU-side carve-out — intentional, do not change. [[Dankclaw System Specs]]
|
||||||
|
|
||||||
**Load / crash diagnosis**
|
**Load / crash diagnosis**
|
||||||
|
|
||||||
- Model load is slow (minutes cold); health returns 503 while loading; do
|
- Model load is slow (minutes cold) and health returns 503 while it runs: do NOT kill the process,
|
||||||
not kill the process. `deploy.sh` waits up to 20 minutes (`LLM_WAIT`). On
|
`deploy.sh` waits up to 20 minutes (`LLM_WAIT`). On halogen the FIRST request after a (re)start
|
||||||
halogen, the FIRST request after a (re)start can go silent ~45–60 s
|
can go silent for ~45–60 s (`engine_unresponsive`, 503s) and then self-recovers — do not restart
|
||||||
(health `engine_unresponsive`, 503s) and then self-recovers — do not
|
over it. [[frank-dragoon]]
|
||||||
restart over it.
|
- Crash-loop procedure: (1) `journalctl -u <svc> -n 100` AND the kernel line via `journalctl -k` (a
|
||||||
- Crash-loop procedure: (1) `journalctl -u <svc> -n 100` + kernel line via
|
segfault at address 8 in `libamdhip64` is a NULL unref in HIP); (2) run the serve script, then the
|
||||||
`journalctl -k` (segfault at address 8 in `libamdhip64` = NULL unref in
|
raw binary — a different outcome isolates the env file; (3) `gdb -batch -ex run -ex "bt 25"
|
||||||
HIP); (2) run serve script, then raw binary — different outcome isolates
|
--args bash scripts/serve-<svc>.sh`, because llama-server buffers stdout and loses the log tail on
|
||||||
the env file; (3) `gdb -batch -ex run -ex "bt 25" --args bash
|
SIGSEGV: trust gdb, not the log; (4) `amdgpu: SVM mapping failed` spam is a symptom, not a root
|
||||||
scripts/serve-<svc>.sh` — llama-server buffers stdout and loses the log
|
cause. [[LLM Performance Diagnosis]]
|
||||||
tail on SIGSEGV; trust gdb, not the log; (4) `amdgpu: SVM mapping failed`
|
- Journal signatures: a client abort logs `W srv stop: cancel task, id_task = N` while a natural stop
|
||||||
spam is a symptom, not a root cause.
|
logs only `slot release` with `truncated = 0`. The server never preempts a running generation;
|
||||||
- Journal signatures: client abort logs `W srv stop: cancel task, id_task =
|
excess tasks queue when slots are busy.
|
||||||
N`; natural stop logs only `slot release` with `truncated = 0`. The server
|
|
||||||
never preempts a running generation; excess tasks queue when slots are
|
|
||||||
busy.
|
|
||||||
|
|
||||||
**Host memory / swap**
|
**Host memory / swap**
|
||||||
|
|
||||||
- Swap resize ordering: create and `swapon` the replacement BEFORE `swapoff`
|
- **Swap resize ordering: create and `swapon` the replacement BEFORE `swapoff` of the old one**, or
|
||||||
of the old one, or live pages must land in RAM that `MemAvailable`
|
live pages must land in RAM that `MemAvailable` overstates. Dankclaw's 64 GiB `/swap.img`
|
||||||
overstates. Dankclaw swap (64 GiB at `/swap.img`, fstab-driven) protects
|
(fstab-driven) protects the co-tenants, not an endpoint with pinned weights — paging the KV pool
|
||||||
the co-tenants, not frank-dragoon — pinned weights don't swap, and paging
|
is worse than the OOM it prevents. Host swap state is invisible to `deploy.sh` (no repo file):
|
||||||
the KV pool is worse than the OOM it prevents.
|
`swapon --show` before reasoning about headroom. [[Swap Configuration]]
|
||||||
- Host swap state is invisible to `deploy.sh` (no repo file) — check
|
- **A sysfs raise is a countdown to the next reboot.** `ttm.pages_limit` defaults to half of RAM, so
|
||||||
`swapon --show` before reasoning about headroom.
|
a manual raise vanished with a power failure, frank-archon's pinned-ledger preflight refused the
|
||||||
- **A sysfs raise is a countdown to the next reboot.** `ttm.pages_limit` defaults
|
start, and `Restart=always` re-ran that refusal 6,919 times in ~20 h — **a unit in
|
||||||
to half of RAM, so the manual raise to 200 GiB (2026-10-05) vanished with the
|
`activating (auto-restart)` that exits 1 in 50 ms with no container and no engine log is a
|
||||||
2026-10-12 23:17 POWER FAILURE on dankserver: the cap fell to 125,988 MiB,
|
PREFLIGHT refusal, not an engine fault.** Repo law now: `scripts/ttm-preflight.sh` raises the knob
|
||||||
frank-archon's pinned-ledger preflight (floor 150,000 MiB) refused the start,
|
to `FRANK_ARCHON_TTM_TARGET_MIB` itself, re-reads, and refuses by name only after a write the
|
||||||
and `Restart=always` re-ran that refusal **6,919 times in ~20 h** — a unit in
|
kernel rejected. Never prescribe a manual sysfs write as the recovery procedure for a service that
|
||||||
`activating (auto-restart)` that exits 1 in 50 ms with no container and no
|
can boot before a human reaches it. [[frank-archon]]
|
||||||
engine log is a PREFLIGHT refusal, not an engine fault. Repo law now:
|
- **Silent RAM corruption outranks every software theory.** frank-archon's long-conversation
|
||||||
`scripts/ttm-preflight.sh` (sourced by `serve-frank-archon.sh`) raises the knob
|
repetition/garbage was marginal DDR5 at 4800 MT/s on four DIMMs, fixed by dropping to 4600.
|
||||||
to `FRANK_ARCHON_TTM_TARGET_MIB` itself, re-reads, and refuses by name only
|
Consumer non-ECC memory raises NO MCE for a silent flip, and a short in-band pattern check
|
||||||
after a write the kernel rejected. Never prescribe a manual sysfs write as the
|
(60 s over 8 GiB of 246 GiB) is not a stability proof — combined, those two instruments said
|
||||||
recovery procedure for a service that can boot before a human reaches it.
|
"ruled out" while the RAM was the cause, and "4800 is the JEDEC floor" is a rating, not a margin
|
||||||
- **Silent RAM corruption outranks every software theory.** frank-archon's
|
guarantee. When the fault is data-shaped (NaN states, garbage tokens) and a software revert does
|
||||||
~40k-token output corruption (2026-10-04/05, repetition/garbage in long
|
not move it, price in the RAM before the runtime — and remember a revert tested under the same
|
||||||
agent conversations) was marginal DDR5 at 4800 MT/s on four DIMMs, fixed
|
marginal hardware is itself confounded. [[log]]
|
||||||
by 4600. Consumer non-ECC memory raises **no MCE** for a silent flip, and
|
|
||||||
a short in-band pattern check (60 s, 8 GiB of 246 GiB) is not a stability
|
|
||||||
proof — combined, those two instruments said "ruled out" while the RAM
|
|
||||||
was the cause. "4800 is the JEDEC floor on 5600-rated DIMMs" is a rating,
|
|
||||||
not a margin guarantee. When the fault is data-shaped (NaN states,
|
|
||||||
garbage tokens) and a software revert doesn't move it, price in the RAM
|
|
||||||
before the runtime — and remember a revert tested under the same marginal
|
|
||||||
hardware is itself confounded. See the runbook's 2026-10-05 section.
|
|
||||||
|
|
||||||
**Runtime-specific guards**
|
**Runtime-specific guards**
|
||||||
|
|
||||||
- **Rootless podman cannot RAISE rlimits, and `--ulimit nofile=1048576`
|
- **Rootless podman cannot RAISE rlimits, so `--ulimit nofile=1048576` dies at container CREATE**,
|
||||||
dies at container create, not at runtime.** runc reports
|
not at runtime: runc reports `error setting rlimit type 7: operation not permitted` (type 7 =
|
||||||
`error setting rlimit type 7: operation not permitted` (type 7 =
|
RLIMIT_NOFILE) when the requested hard limit exceeds the host process's ceiling (524288 on
|
||||||
RLIMIT_NOFILE) when the requested hard limit exceeds the host process's
|
Dankclaw). Inherited-equal limits are fine, so `memlock=-1:-1` works iff the unit grants
|
||||||
hard limit — on Dankclaw that ceiling is 524288. Inherited-equal limits
|
`LimitMEMLOCK=infinity` — systemd is privileged and the value flows through podman to runc without
|
||||||
are fine, so `memlock=-1:-1` works iff the unit grants
|
any raise. Upstream halogen's own run lines pass NO `--ulimit` at all: copy vendor run lines
|
||||||
`LimitMEMLOCK=infinity` (systemd is privileged; the value flows through
|
before adding flags. [[frank-dragoon]]
|
||||||
podman to runc without any raise). Upstream halogen's own run lines pass
|
- **A GGUF's custom tensor type picks the runtime — read the histogram before promising a file to a
|
||||||
NO `--ulimit` at all — copy vendor run lines before adding flags
|
loader.** `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is ftype 115 over 311 tensors of type 103
|
||||||
(measured 2026-10-05 on the frank-dragoon cutover).
|
(`GGML_TYPE_Q8_0_ROCMFPX`), so stock llama.cpp and gufo refuse it at load and the fork was not a
|
||||||
- **Kairic Edge** (if ever re-deployed): the pinned CK commit needs
|
preference but the only door. Counting tensor types out of the GGUF KV section takes one python
|
||||||
`third_party/composable_kernel/include/ck/config.h` generated manually
|
read and settles a runtime argument in minutes. Build identity matters as much as the fork: the
|
||||||
(untracked; `git clean` removes it), and its greedy-argmax fastpath rejects
|
2026-08-25 ROCmFPX builds SIGSEGV in `ggml_compute_forward_set_rows` on this same file, so
|
||||||
non-greedy requests and reasoning budgets. Re-test chat before production.
|
`scripts/build-rocmfpx.sh` pins commit `36e9acd` by SHA. [[GGUF Compatibility]]
|
||||||
- **FastMTP sidecar**: build must be commit `4df29be4` + the HauhauCS patch;
|
- **Speculative-decode flags move as a SET.** Handing the fork `--model-draft` with no `--spec-type`
|
||||||
stock binaries fail the draft load with a tensor-shape mismatch.
|
(the naive way to build a "no MTP" bench arm) segfaulted at load, core dumped, no diagnostic. A
|
||||||
- **A GGUF's custom tensor type picks the runtime — read the histogram before
|
baseline arm must strip `--model-draft` AND every `--spec-draft-*` AND `--spec-type`, or it
|
||||||
promising a file to a loader.** `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is
|
measures a crash instead of the absence of drafting. [[frank-zealot]]
|
||||||
ftype 115 with 311 tensors of type 103 (`GGML_TYPE_Q8_0_ROCMFPX`); stock
|
- **An orphaned scratch server taxes the whole host.** A bench child left alive by a client-side
|
||||||
llama.cpp and gufo refuse it at load, so ROCmFPX was not a preference but
|
exception kept ~26 GiB of weights resident on a 121 GiB box and pushed `available` to 3 GiB, so
|
||||||
the only door. Counting tensor types out of the GGUF KV section takes one
|
every later rate was suspect. Record the child pid, kill it in a `finally`, then prove `free` came
|
||||||
python read and settles a runtime argument in minutes. The build identity
|
back before believing a number — and kill by pid, never `pkill -f llama-server`.
|
||||||
matters as much as the fork: the 2026-08-25 ROCmFPX builds SIGSEGV in
|
[[LLM Bench Windows and Measurement]]
|
||||||
`ggml_compute_forward_set_rows` on this same file, so
|
- Retired-runtime traps that can come back: **Kairic Edge** needs
|
||||||
`scripts/build-rocmfpx.sh` pins commit `36e9acd` by SHA.
|
`third_party/composable_kernel/include/ck/config.h` generated manually (untracked, so `git clean`
|
||||||
- **Speculative-decode flags move as a SET.** Handing the ROCmFPX fork
|
removes it) and its greedy-argmax fastpath rejects non-greedy requests and reasoning budgets, so
|
||||||
`--model-draft` with no `--spec-type` (the naive way to build a "no MTP"
|
re-test chat before production; the **FastMTP sidecar** must be commit `4df29be4` + the HauhauCS
|
||||||
bench arm) segfaulted at load, core dumped, no diagnostic (2026-10-02). A
|
patch, since stock binaries fail the draft load with a tensor-shape mismatch.
|
||||||
baseline arm must strip `--model-draft` AND every `--spec-draft-*` AND
|
|
||||||
`--spec-type`, or it measures a crash instead of the absence of drafting.
|
|
||||||
- **An orphaned scratch server taxes the whole host.** A bench child left
|
|
||||||
alive by a client-side exception kept ~26 GiB of weights resident on a
|
|
||||||
121 GiB box, pushed `available` to 3 GiB, and every later rate was suspect.
|
|
||||||
Record the child pid and kill it in a `finally`, then prove `free` came back
|
|
||||||
before believing a number. Kill by pid — never `pkill -f llama-server`,
|
|
||||||
which also hits frank-zealot.
|
|
||||||
|
|
||||||
**llama.cpp serving**
|
**llama.cpp serving**
|
||||||
|
|
||||||
- **MTP is a property of the FILE, not of the flags.** The `-mtp` GGUF build
|
- **MTP is a property of the FILE, not of the flags.** The `-mtp` export carries the head as
|
||||||
carries the head as `blk.<n>.nextn.*` (`qwen35.nextn_predict_layers = 1`);
|
`blk.<n>.nextn.*` (`qwen35.nextn_predict_layers = 1`); `--spec-type draft-mtp` sets
|
||||||
`--spec-type draft-mtp` then sets `load_mtp`, and the draft context runs on
|
`load_mtp` and the draft context runs on the MAIN model's weights (`shares_model`), so there
|
||||||
the MAIN model's weights (`shares_model`), so there is no second copy of a
|
is no second copy of a 12 GiB model in VRAM and no `-md` argument. The plain build has no MTP
|
||||||
12 GiB model in VRAM and no `-md` argument. The plain build has no MTP
|
tensors at all, and pointing `--model-draft` at that file loads it twice. [[frank-zealot]]
|
||||||
tensors at all, and pointing `--model-draft` at the file would load it twice.
|
- **What can be chained: ONE context-backed drafter plus any number of draftless ngram types.**
|
||||||
- **What can be chained, what cannot, and what a chain costs on a hybrid.**
|
`--spec-type a,b` chains in a fixed internal order (CLI order irrelevant) and the gate is
|
||||||
`--spec-type a,b` builds every enabled implementation and chains drafting
|
`ctx_dft`: the ngram family is added unconditionally while `draft-mtp`, `draft-dflash`,
|
||||||
through them, in a fixed internal order (ngram types first, then the
|
`draft-dspark` and `draft-eagle3` all need a draft CONTEXT. So `draft-dflash,ngram-mod` is
|
||||||
drafters — CLI order is irrelevant). The gate is `ctx_dft`: **draftless**
|
legal (measured 2.6x the drafter alone on prompt-echo traffic) but `draft-mtp,draft-dflash`
|
||||||
types (the ngram family) are added unconditionally, while `draft-mtp`,
|
dies at load — *"context type MTP requested but model doesn't contain MTP layers"*. **A LEGAL
|
||||||
`draft-dflash`, `draft-dspark` and `draft-eagle3` all require a draft
|
chain can still lose on a recurrent model:** `draft-mtp,ngram-mod` on frank-dragoon's qwen35
|
||||||
CONTEXT. So a chain may hold one context-backed drafter plus any number of
|
hybrid measured −15 % prose / −7 % block-copy, because adding the ngram layer logs
|
||||||
ngram types — `draft-dflash,ngram-mod` is legal and measured (2.6x the
|
`recurrent state rollback is not compatible with 'ngram-mod' - disabling rollback support` and
|
||||||
drafter alone on prompt-echo traffic, byte-identical to it on prose at the
|
MTP verification depends on that rollback. Measure chains per architecture, never by analogy.
|
||||||
fleet's 48/64 window) — but `draft-mtp,draft-dflash` dies at load: *"context
|
[[frank-dragoon]]
|
||||||
type MTP requested but model doesn't contain MTP layers"*, because MTP
|
- **A drafter's KV scales with the TARGET's context, and its window policy decides the price.**
|
||||||
builds its context against whatever model the draft context holds. Two
|
The draft context is built at `llama_n_ctx(ctx_tgt)` with no flag to cap it, so a drafter
|
||||||
context-backed types cannot share one process, and MTP's head only loads
|
declaring NO sliding window caches the whole window on `--spec-draft-device` and can OOM the
|
||||||
when MTP **is** the draft model. **A LEGAL chain can still lose on a
|
first draft graph capture AFTER `/health` has answered. Read `sliding_window` out of the
|
||||||
recurrent model:** `draft-mtp,ngram-mod` on frank-dragoon's qwen35 hybrid
|
drafter's GGUF header before sizing, and prefer a skewed `-ts` over a context cut.
|
||||||
measured −15 % prose / −7 % block-copy (2026-10-03) — adding the ngram layer
|
[[Llama Cpp Multi-Device Splits]]
|
||||||
logs `recurrent state rollback is not compatible with 'ngram-mod' -
|
- **A draft context bills its own f16 KV over the WHOLE POOL, not the slot**, plus its own
|
||||||
disabling rollback support`, and MTP verification depends on that rollback,
|
compute buffers, so on a single-card endpoint that squeeze can cost more than the drafter
|
||||||
so acceptance drops on ALL traffic even where the matcher fires. Measure
|
wins. The MTP question is never "does the file carry the head" but what draft KV + buffers
|
||||||
chains per architecture, never by analogy.
|
cost against what the batch knobs leave for compute. Measure the arm, never the flag.
|
||||||
- **A drafter's KV scales with the TARGET's context, and its window policy
|
[[frank-zealot]]
|
||||||
decides the price.** The draft context is built at `llama_n_ctx(ctx_tgt)` with
|
- **A quant's scale overhead belongs in its byte rate; doubling a measured figure to "get the
|
||||||
no flag to cap it, so a drafter declaring NO sliding window caches the whole
|
f16 number" invents 6.25 %.** q8_0 is 1.0625 B/element (32 codes + one 2-byte scale per
|
||||||
window on `--spec-draft-device` and can OOM the first draft graph capture
|
32-block), f16 is exactly 2.0 ⇒ the ratio is **1.882×**, not 2×. Derive byte rates from element
|
||||||
AFTER `/health` has answered. Read `sliding_window` out of the drafter's GGUF
|
count × block layout, then check them against the boot ledger — this repo carried an invented
|
||||||
header before sizing, and prefer a skewed `-ts` over a context cut. Worked
|
2× number for weeks. Current KV math: [[frank-zealot]]
|
||||||
example: [[frank-medium-llamacpp-dflash2]].
|
- **`--ubatch-size` has an OPTIMUM, not a plateau, and the curve is BACKEND-SPECIFIC — never
|
||||||
- **A draft context bills its own f16 KV over the WHOLE POOL, and on a
|
carry it across hosts.** Its compute buffers land about TWICE as heavy on the card that holds
|
||||||
single-card endpoint that squeeze can cost more than the drafter wins.** With
|
the output layers and the drafter, so a split balanced at one batch is unbalanced at another:
|
||||||
`--parallel N` the drafter pays the pool, not the slot, plus its own compute
|
re-measure the balance after any batch change. The logical `--batch-size` is nearly free; a
|
||||||
buffers. So the MTP question is never "does the file carry the head" but "what
|
large-batch allocation can fail on BUFFER CONTIGUITY while totals still fit, which no totals
|
||||||
do draft KV + buffers cost against what the batch knobs leave for compute" —
|
model can see. A prompt shorter than the ubatch cannot see the knob at all — that is the
|
||||||
a per-shape answer. Measure the arm, never the flag; the booted arm table is
|
control that proves the knob, not the traffic, moved the number. The 2× R9700 split peaks at
|
||||||
on [[frank-zealot]].
|
512; the CUDA 3090 has no interior optimum up to 2048. When a bench IS ordered, A/B it by
|
||||||
- **A quant's scale overhead belongs in its byte rate; doubling a measured
|
restarting the SERVICE through a transient `FRANK_<SVC>_ENV_FILE` drop-in, never a scratch
|
||||||
figure to "get the f16 number" invents 6.25 %.** q8_0 is 1.0625 B/element
|
port: one card holds one server. Ladders: [[frank-medium-llamacpp-dflash2]] and
|
||||||
(32 codes + one 2-byte scale per 32-block), f16 is exactly 2.0 ⇒ the ratio is
|
[[frank-zealot]].
|
||||||
**1.882×**, not 2×. Derive byte rates from element count × block layout, then
|
- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same size** (a DeltaNet
|
||||||
check them against the boot ledger — this repo carried an invented 2× number
|
boundary and a full-attention boundary can differ by ~1 GiB each with its own KV), so balance
|
||||||
for weeks. Current KV math: [[frank-zealot]].
|
by measurement. An affine fit reproduces the deployed point and overstates the gaps in between.
|
||||||
- **`--ubatch-size` has an OPTIMUM, not a plateau, and the curve is
|
[[Llama Cpp Multi-Device Splits]]
|
||||||
BACKEND-SPECIFIC — never carry it across hosts.** Its compute buffers land
|
- **`ngram-mod` prints no startup banner.** The load log names `draft-dflash` and nothing for the
|
||||||
about TWICE as heavy on the card that holds the output layers and the drafter,
|
matcher, so a chain can look unengaged while it works. Identify it behaviourally (`draft_n` > 0
|
||||||
so a split balanced at one batch is unbalanced at another: re-measure the
|
only where text repeats) or by the per-task `draft acceptance = ... mean len = ...` line. It
|
||||||
balance after any batch change. The logical `--batch-size` is nearly free.
|
costs no VRAM (a 64 MiB host-RAM hash table) and its window decides which traffic it can see:
|
||||||
A large-batch allocation can fail on BUFFER CONTIGUITY while totals still fit
|
too short and it proposes on ordinary prose fragments, costing verification without acceptance.
|
||||||
— no totals model can see that. Ladders: [[frank-medium-llamacpp-dflash2]]
|
[[Llama Cpp Multi-Device Splits]]
|
||||||
(R9700 split) and [[frank-zealot]] (CUDA 3090).
|
- **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the per-slot number is
|
||||||
- **On the CUDA 3090 that same knob has no interior optimum up to 2048.** A
|
the contract — not `ctx-size`.** `n_ctx_slot = CTX_SIZE / PARALLEL`, and llama.cpp refuses a
|
||||||
prompt shorter than the ubatch cannot see the knob at all — that is the
|
prompt longer than the SLOT with a 400 even when the pool has room, so `models.yml`
|
||||||
control that proves the knob, not the traffic, moved the number. When a bench
|
`contextWindow` MUST follow every reslice. Both slot directions cost or buy window, never
|
||||||
IS ordered, A/B it by restarting the SERVICE through a transient
|
pool; shorter slots decode slightly faster. Measured reslices: [[frank-zealot]].
|
||||||
`FRANK_ZEALOT_ENV_FILE` drop-in, never a scratch port: one card holds one
|
- **Price a VRAM budget from two boots, never from a formula — and an mmproj costs ~1.5× its file
|
||||||
server. Numbers: [[frank-zealot]].
|
size** (the encoder's activation and compute buffers ride with it). Subtracting two booted rows
|
||||||
- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same
|
isolates it exactly; extrapolating weights+KV+"a bit for buffers" is what leaves 1.1 GiB free
|
||||||
size** (a DeltaNet boundary and a full-attention boundary can differ by ~1 GiB
|
when you meant 2.7. Then solve `KV ≤ card − non-KV − headroom` and round to a multiple of
|
||||||
each with its own KV), so balance by measurement. An affine fit reproduces the
|
`PARALLEL × 4096`. **The projector is NOT billed in llama.cpp's own fit check**: the engine
|
||||||
deployed point and overstates the gaps in between.
|
prints its worst case and fits WITHOUT it, so price a shape from settled `nvidia-smi` after a
|
||||||
[[frank-medium-llamacpp-dflash2]].
|
real boot — the projector, and any drafter context, lands after `/health` would already have
|
||||||
- **ngram-mod prints no startup banner.** The load log shows the drafter's
|
answered (same class as the DSpark drafter OOM). Two-boot table: [[frank-zealot]].
|
||||||
`adding speculative implementation 'draft-dflash'` and nothing at all for the
|
- **`images` and `image_url` are NOT interchangeable: only the content part reaches the model.**
|
||||||
matcher, so a chain can look unengaged while it is working. Identify it
|
`--mmproj` loads, `usage.prompt_tokens` rises, and the model still invents an answer if the
|
||||||
behaviourally (`draft_n` > 0 only where text repeats) or by the per-task
|
client used the legacy top-level `images` field on `/v1/chat/completions` — frank-zealot
|
||||||
`draft acceptance = ... mean len = ...` line. It costs no VRAM (a 64 MiB
|
answered "Brown." for a solid-red PNG that way and "Red." through the OpenAI `image_url` part.
|
||||||
host-RAM hash table) and its window decides which traffic it can see: too
|
The failure is silent because a hallucinated color is a plausible answer; probe with a solid
|
||||||
short and it proposes on ordinary prose fragments, which costs verification
|
color and expect an exact word. [[frank-zealot]]
|
||||||
without acceptance.
|
- **`--jinja` is what `tools` needs.** A subagent died with `500 tools param requires --jinja
|
||||||
- **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the
|
flag` on the ik_llama.cpp runtime that served frank-dragoon: every tool-using agent on that
|
||||||
per-slot number is the contract — not `ctx-size`.** `n_ctx_slot =
|
endpoint was broken and the failure read like a model fault. Mainline llama.cpp defaults
|
||||||
CTX_SIZE / PARALLEL`, and llama.cpp refuses a prompt longer than the SLOT with
|
`--jinja` on and the serve scripts pin it; halogen also accepts `tools` and returns
|
||||||
a 400 even when the pool has room, so `models.yml` `contextWindow` must follow
|
`tool_calls`. Check tool support before routing a role at an endpoint.
|
||||||
every reslice. Both slot directions cost or buy window, never pool; shorter
|
- **A model's own template can REJECT the harness's thinking level.** Qwen3.8's template accepts
|
||||||
slots decode slightly faster (less attention to scan, one fewer recurrent
|
only `xhigh`, `medium` and `low` for `reasoning_effort` and raises on anything else — HTTP 500,
|
||||||
state). Measured reslices: [[frank-zealot]].
|
not a clamp — so the harness's levels must be folded onto that set in `models.yml`
|
||||||
- **Price a VRAM budget from two boots, never from a formula — and an mmproj
|
(`compat.reasoningEffortMap`). A role whose level is unmapped fails on its FIRST request and
|
||||||
costs ~1.5× its file size** (the encoder's activation and compute buffers ride
|
the failure names Jinja, not the model: every subagent on `frank-zealot` died with
|
||||||
with it). Subtracting two booted rows isolates it exactly; extrapolating
|
`Unexpected reasoning effort high` until `high: xhigh` was added. Halogen accepts the wider
|
||||||
weights+KV+"a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7.
|
set, so the two entries legitimately differ.
|
||||||
Then solve `KV ≤ card − non-KV − headroom` and round to a multiple of
|
- **Thinking tokens count against `max_tokens`, so a short-budget vision probe reads as
|
||||||
`PARALLEL × 4096`. The two-boot table: [[frank-zealot]].
|
blindness.** A probe at `max_tokens: 64` returned `content: ""` with `finish_reason: "length"`
|
||||||
- **`images` and `image_url` are NOT interchangeable: only the content part
|
and half a thought in `reasoning_content` while the image HAD reached the model (322 prompt
|
||||||
reaches the model.** `--mmproj` loads, `usage.prompt_tokens` rises, and the
|
tokens vs 42 for the text alone). Budget ≥512 for any thinking model you probe for vision.
|
||||||
model still invents an answer if the client used the legacy top-level
|
- **The FIRST request after a boot is a warmup artifact**, and the 3× spread on one shape makes
|
||||||
`images` field on `/v1/chat/completions` — frank-zealot answered "Brown." for
|
graph-capture cost read as a model fault. Benchmark the second request or later.
|
||||||
a solid-red PNG that way and "Red." through the OpenAI `image_url` part. The
|
[[frank-zealot]]
|
||||||
failure is silent because a hallucinated color is a plausible answer. Probe
|
- **A fork build hides its own dtype laws.** Vendor builds carry their own compile options, so
|
||||||
vision with a solid color and expect an exact word.
|
prove a build's KV behaviour from the KV buffer line in the boot ledger and its
|
||||||
- **`--jinja` is what `tools` needs.** On 2026-09-16 a subagent died with
|
`CMakeCache.txt` — never from the build flags you remember passing. [[frank-zealot]]
|
||||||
`500 tools param requires --jinja flag`, from the ik_llama.cpp runtime that
|
- **A llama.cpp endpoint is invisible to Prometheus unless `--metrics` is passed**: without it
|
||||||
then served frank-dragoon: every tool-using agent on that endpoint was broken
|
the engine answers **501 Not Implemented** on `/metrics`, so the scrape fails while `/health`
|
||||||
and the failure read like a model fault. Mainline llama.cpp 0.4.1 defaults
|
answers 200 and `LlmEndpointDown` fires beside a healthy service — that is how `frank-zealot`
|
||||||
`--jinja` on and the serve script pins it; halogen also accepts `tools` and
|
sat red across four payload generations. halogen serves `/metrics` natively, so the asymmetry
|
||||||
returns `tool_calls`. Check tool support before routing a role at an endpoint.
|
reads as "only the 3090 is broken". Diagnose with `lastError` from `/api/v1/targets`, not the
|
||||||
- **Quantized KV on CUDA needs `GGML_CUDA_FA_ALL_QUANTS=ON`** — the same
|
alert list, and note Alertmanager has no persistence: a reboot re-stamps `startsAt` on an alert
|
||||||
option and the same ~12x prefill collapse described in the ROCm note above.
|
that has fired for weeks. [[Observability]]
|
||||||
f16 KV needs nothing; frank-zealot ran f16 for that reason until
|
- **A leftover bench drop-in silently hijacks the next deploy, and no health gate sees it.** The
|
||||||
2026-09-17, when `build-cuda` was rebuilt with the option ON so the
|
sanctioned bench-window method (a transient `Environment=FRANK_<SVC>_ENV_FILE=/tmp/<arm>.env`
|
||||||
endpoint could take q8_0 KV.
|
drop-in sourcing the repo env) outlives the bench unless retired: a leftover
|
||||||
- **llama-bench separates a LIST with `/` and SWEEP values with `,`.** Both
|
`frank-zealot.service.d/ptq1.conf` made the cutover deploy boot a **different model at a
|
||||||
`-dev` and `-ts` follow this: `-dev CUDA0,Vulkan1` is two *single-device*
|
different shape** while `/health` answered 200, `/v1/models` echoed the alias (a flag, not a
|
||||||
configurations, never one two-device run, and `-ts 0.85,0.15` is two sweep
|
fact) and the Playwright gate passed 33/33. After deploying any recently-benched LLM service:
|
||||||
values each becoming `[0.85, 0, …]` / `[0.15, 0, …]`, i.e. device 0 takes
|
`systemctl show <svc> -p DropInPaths` (must be empty), then
|
||||||
everything. Write `-dev A/B -ts x/y`. This artifact produced a wrong fleet
|
`tr '\0' ' ' </proc/$(pidof llama-server)/cmdline`, then the boot ledger — and retire drop-ins
|
||||||
conclusion on 2026-09-16 — "CUDA + Vulkan cannot serve one model", with two
|
to `~/.AGENTTRASH/<date>-<topic>/` + `daemon-reload`. [[frank-zealot]]
|
||||||
bogus OOMs that were really "all 111 GiB on the 24 GiB card" — and the bench
|
- **A bench verdict that never reaches the repo is indistinguishable from a rogue arm.** That
|
||||||
table's own `dev` column was printing the truth the whole time. Any
|
same arm served production for 24 minutes with real numbers (~48 t/s decode, 801 t/s prefill at
|
||||||
placement-dependent number needs an independent check: sample the other
|
141k) that lived only in `/tmp` env files and journald, so the next agent read the surviving
|
||||||
|
drop-in as an unfinished experiment and reported a hijack. Write the measurement into the
|
||||||
|
endpoint page and the arm script's header in the session that takes it — `/tmp` and the journal
|
||||||
|
are not records.
|
||||||
|
- **`llama-bench` separates a LIST with `/` and SWEEP values with `,`** — for both `-dev` and
|
||||||
|
`-ts`. `-dev CUDA0,Vulkan1` is two *single-device* configurations, never one two-device run,
|
||||||
|
and `-ts 0.85,0.15` is two sweep values each putting everything on device 0. Write
|
||||||
|
`-dev A/B -ts x/y`. This artifact produced a wrong fleet conclusion ("CUDA + Vulkan cannot serve
|
||||||
|
one model") from two bogus OOMs, and the bench table's own `dev` column was printing the truth
|
||||||
|
the whole time. Any placement-dependent number needs an independent check: sample the other
|
||||||
device's memory (`nvidia-smi`) while the run is loaded.
|
device's memory (`nvidia-smi`) while the run is loaded.
|
||||||
- **A model's own template can REJECT the harness's thinking level.** Qwen3.8's
|
|
||||||
template accepts only `xhigh`, `medium` and `low` for `reasoning_effort` and
|
|
||||||
raises on anything else — HTTP 500, not a clamp — so the harness's levels
|
|
||||||
must be folded onto that set in `models.yml` (`compat.reasoningEffortMap`).
|
|
||||||
A role whose level is unmapped fails on its FIRST request, and the failure
|
|
||||||
names Jinja, not the model: every subagent on `frank-zealot` died with
|
|
||||||
`Unexpected reasoning effort high` until `high: xhigh` was added on
|
|
||||||
2026-09-16. Halogen accepts the wider set, so the two entries legitimately
|
|
||||||
differ.
|
|
||||||
- **The vision projector is NOT billed in llama.cpp's own fit check.** The
|
|
||||||
engine prints `[mtmd] estimated worst-case memory usage of mmproj is
|
|
||||||
873.10 MiB` and then fits WITHOUT it: frank-zealot's Bonsai-2 boot projected
|
|
||||||
21,439 MiB and settled at 22,664 (2026-10-17). Price a 3090-class shape from
|
|
||||||
settled `nvidia-smi` after a real boot, never from
|
|
||||||
`model + context + compute` — the projector (and any drafter context) lands
|
|
||||||
after `/health` would already have answered. Same failure class as the
|
|
||||||
DSpark drafter OOM.
|
|
||||||
- **The FIRST request after a boot is a warmup artifact.** Measured on
|
|
||||||
Bonsai-2/3090: 2.6 t/s prefill and 37.7 t/s decode on request #1, then
|
|
||||||
1,258 t/s prefill and 54.7 t/s (per stream, two concurrent) on the next — a
|
|
||||||
3× spread on the same shape. Benchmark the second request or later, or the
|
|
||||||
graph-capture cost reads as a model fault.
|
|
||||||
- **Thinking tokens count against `max_tokens`, so a short-budget vision probe
|
|
||||||
reads as blindness.** The Bonsai-2/`image_url` probe at `max_tokens: 64`
|
|
||||||
returned `content: ""` with `finish_reason: "length"` and half a thought in
|
|
||||||
`reasoning_content` — the image HAD reached the model (322 prompt tokens vs
|
|
||||||
42 for the text alone). At 512 tokens the same image answers "Red". Budget
|
|
||||||
≥512 for any thinking model you probe for vision.
|
|
||||||
- **Quantized KV is not the only dtype law a fork build can hide.** Vendor
|
|
||||||
builds carry their own compile options: frank-zealot's PrismML prebuilt
|
|
||||||
served `--cache-type-v q8_0` under flash attention at the exact 34,816
|
|
||||||
B/token (the `GGML_CUDA_FA_ALL_QUANTS` door), so a build decision must be
|
|
||||||
proven by the KV buffer line in the boot ledger, not by the build flags you
|
|
||||||
remember passing.
|
|
||||||
- **A llama.cpp endpoint is invisible to Prometheus unless `--metrics` is passed.**
|
|
||||||
Without it the engine answers **501 Not Implemented** on `/metrics`, so the scrape
|
|
||||||
fails while `/health` answers 200 and `LlmEndpointDown` fires beside a healthy
|
|
||||||
service — that is how `frank-zealot` sat red across four payload generations
|
|
||||||
(the oldest retained sample of `up{job="frank-zealot"}`, 2026-10-11, is already
|
|
||||||
0). halogen serves `/metrics` natively, so the asymmetry reads as "only the 3090
|
|
||||||
is broken". Diagnose with `lastError` from `/api/v1/targets`, not the alert list,
|
|
||||||
and note Alertmanager has no persistence: a reboot re-stamps `startsAt` on an
|
|
||||||
alert that has been firing for weeks.
|
|
||||||
- **A leftover bench drop-in silently hijacks the next deploy, and no health
|
|
||||||
gate sees it.** The sanctioned bench-window method — a transient
|
|
||||||
`Environment=FRANK_<SVC>_ENV_FILE=/tmp/<arm>.env` drop-in, the arm env
|
|
||||||
`source`ing the repo env and then overriding it — outlives the bench unless
|
|
||||||
it is retired. On 2026-10-18 a leftover `frank-zealot.service.d/ptq1.conf` —
|
|
||||||
the arm Frank had ordered at 02:36 and measured from 03:14 — made the cutover
|
|
||||||
deploy boot a **different model at a different shape** (3 slots of 133,888,
|
|
||||||
not 3 × 262,144) while `/health` answered 200, `/v1/models` echoed the alias
|
|
||||||
(a flag, not a fact) and the Playwright gate passed 33/33. After deploying any
|
|
||||||
recently-benched LLM service: `systemctl show <svc> -p DropInPaths` (must be
|
|
||||||
empty), then `tr '\0' ' ' </proc/$(pidof llama-server)/cmdline`, then the boot
|
|
||||||
ledger — and retire drop-ins to `~/.AGENTTRASH/<date>-<topic>/` +
|
|
||||||
`daemon-reload`.
|
|
||||||
- **A bench verdict that never reaches the repo is indistinguishable from a
|
|
||||||
rogue arm.** Same window: PTQ1_0 served production for 24 minutes with real
|
|
||||||
numbers (~48 t/s decode, 801 t/s prefill at 141k), but the figures lived only
|
|
||||||
in `/tmp` env files and journald, so the next agent on the host read the
|
|
||||||
surviving drop-in as an unfinished experiment and reported a hijack. Write the
|
|
||||||
measurement into the runbook and the arm script's header in the session that
|
|
||||||
takes it — `/tmp` and the journal are not records.
|
|
||||||
|
|
||||||
**Ops**
|
**Ops**
|
||||||
|
|
||||||
- Never `pkill -f` a pattern that appears in your own command line — it kills
|
- Never `pkill -f` a pattern that appears in your own command line — it kills your ssh session.
|
||||||
your ssh session. Use `pkill -x <exact-name>`.
|
Use `pkill -x <exact-name>`.
|
||||||
- Both hosts allow passwordless sudo for the deploy user. Use plain `sudo` in
|
- Both hosts allow passwordless sudo for the deploy user. Use plain `sudo` in committed scripts
|
||||||
committed scripts (no `echo pw | sudo -S`); `FRANK_SUDO_PW` covers hosts
|
(no `echo pw | sudo -S`); `FRANK_SUDO_PW` covers hosts without it.
|
||||||
without it.
|
- **ComfyUI on Dankclaw runs with GPU access** (`/dev/kfd` + `/dev/dri` via `AddDevice`,
|
||||||
- ComfyUI on Dankclaw runs with **GPU access** (`/dev/kfd` + `/dev/dri` via
|
`GroupAdd=keep-groups`, plus the udev rule that makes those nodes world-writable — user-unit
|
||||||
`AddDevice`, `GroupAdd=keep-groups`, plus the udev rule that makes those nodes
|
spawns carry no supplementary groups) and must NOT pass `--cpu`: the gfx1151 ROCm wheels exist
|
||||||
world-writable — user-unit spawns carry no supplementary groups). It must NOT
|
and a stale device-less container was found restarted 9,146 times. [[ComfyUI]]
|
||||||
pass `--cpu`: the gfx1151 ROCm wheels exist and the unit's `--cpu` era ended
|
- SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its ports off the gluetun
|
||||||
2026-09-16, when a stale device-less container was found restarted 9,146
|
service. [[SearXNG]] [[Gluetun]]
|
||||||
times (`RuntimeError: No HIP GPUs are available`).
|
- **Dankclaw's kernel command line is a repo file.** It boots **`iommu=pt`** since 2026-10-18,
|
||||||
- SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its
|
tracked in `dankclaw/host/grub`, and moves ONLY through `scripts/set-dankclaw-bootline.sh`
|
||||||
ports off the gluetun service.
|
(diff, install, `update-grub`, verify `grub.cfg`, `--reboot`) — never by hand-editing
|
||||||
- Dankclaw boots **`iommu=pt`** since 2026-10-18. The boot line is tracked in
|
`/etc/default/grub`. It replaced `amd_iommu=off`, and the local A/B on byte-identical traffic
|
||||||
`dankclaw/host/grub` and moves ONLY through `scripts/set-dankclaw-bootline.sh`
|
says identity mode costs nothing, so the XDNA2 NPU returned for free.
|
||||||
(diff, install, `update-grub`, verify `grub.cfg`, `--reboot`) — never by
|
[[dankclaw-iommu-translation-off]]
|
||||||
hand-editing `/etc/default/grub`. It replaced `amd_iommu=off` (2026-09-11),
|
- **Rootless Quadlet containers run with `--replace --rm`**: `podman logs` covers only the CURRENT
|
||||||
adopted on the vendor's 13–16 % prefill figure and never measured here: the
|
instance; history across recreations lives in `journalctl --user -u <name>.service`. Read
|
||||||
local A/B on byte-identical traffic says identity mode costs **nothing**
|
rootless storage WITHOUT sudo — it is per-user storage, so `sudo podman logs` finds nothing.
|
||||||
(+2.0 %, 1,616 t/s cold prefill), so the XDNA2 NPU returned for free —
|
[[Quadlet Lifecycle]]
|
||||||
`amdxdna` binds `c6:00.1` (PASID width 10, identity group 26) and the old
|
- **A dead rootless podman pause process breaks ALL container management on the host, and
|
||||||
`err=-19` only ever asked for an enabled IOMMU. **AMD PerfOpt** (iommu tree
|
`podman system migrate` panics in 5.7.0.** Every `podman` command fails with
|
||||||
2026-09-24, Linux 7.4) still skips the page walk outright for the privileged
|
`invalid internal status`, every start fails binding the bridges
|
||||||
iGPU, but it arms only in the identity domain and 26.04 caps at `7.0.0-38`.
|
(`failed to bind udp listener on 10.89.x.1:53`), every `Restart=always` quadlet loops — while
|
||||||
- Rootless Quadlet containers run with `--replace --rm`: `podman logs` covers
|
container storage stays intact. What is gone is `/run/user/<uid>/libpod/tmp/alive` (a UUID
|
||||||
only the CURRENT instance; history across recreations lives in
|
token) and the rootless netns; deleting the stale token is not enough. Repair: **reboot**
|
||||||
`journalctl --user -u <name>.service`. Read rootless storage WITHOUT sudo
|
(`/run/user` clears and the quadlets return on their own — check `Linger=yes` and
|
||||||
(per-user storage — `sudo podman logs` finds nothing).
|
`WantedBy=default.target` first). The 2026-09-16 occurrence had no OOM in `journalctl -k`:
|
||||||
- **A dead rootless podman pause process breaks ALL container management on the
|
treat the pause process as host state that can die for its own reasons. [[Quadlet Lifecycle]]
|
||||||
host, and `podman system migrate` panics in 5.7.0.** Symptoms: every `podman`
|
- **A container cannot reach a `Network=host` service at the host's LAN IP**: netavark DNATs
|
||||||
command fails with `invalid internal status … could not find any running
|
*published* ports only, so the host bind and the `10.89.1.1` gateway are both refused from a
|
||||||
process`, every container start fails binding podman's bridge addresses
|
bridge container while the published port answers. Use `host.containers.internal`
|
||||||
(`Error starting server failed to bind udp listener on 10.89.x.1:53: Cannot
|
(169.254.1.2). This 502'd `hermes.franksdank.cloud` until its Caddyfile upstream changed.
|
||||||
assign requested address`), and every `Restart=always` quadlet loops. Container
|
[[Container Host Reachability]]
|
||||||
storage is intact; what is gone is `/run/user/<uid>/libpod/tmp/alive` (a UUID
|
- **A `*.gguf` filter misses mradermacher's payloads: large quants ship as `.partNofM`
|
||||||
token, not a pid) and the rootless netns holding the bridges — deleting the
|
fragments** while the smaller rungs are single files, so filtering `endswith(".gguf")` makes a
|
||||||
stale token is not enough, and `podman system migrate` segfaults in
|
whole quant class look absent. Fetch the parts, verify each against the Hub's LFS sha256,
|
||||||
`teardownNetworkBackend`. Repair: **reboot** (`/run/user` clears and all 28
|
concatenate in numeric order. [[Model Swap and Rollback]]
|
||||||
quadlets return on their own — check `Linger=yes` and `WantedBy=default.target`
|
- **A fetch script must `-f` and verify, or it reports success on an access error.**
|
||||||
first). Seen 2026-09-16, hours into heavy GPU work on the same host — but the
|
`curl -o FILE URL` exits 0 on HTTP 401/403 and writes the error body as the "artifact": a gated
|
||||||
trigger is NOT established (`journalctl -k` recorded no OOM kill; the kernel
|
repo yielded three 153-byte shards with identical sha256 and a cheerful `DOWNLOAD_DONE` in 20
|
||||||
had no other word on it). Treat the pause process as host state that can die
|
seconds for 104 GiB. Use `-f`/`--fail-with-body`, compare each file's byte count to the Hub's
|
||||||
for its own reasons, and reach for the reboot when it does.
|
declared size, and check the LFS sha256 (`scripts/download-frank-medium-gguf.sh` does all three;
|
||||||
- **A container cannot reach a `Network=host` service at the host's LAN IP.**
|
ad-hoc fetches tend not to). Gated repos 403 even with the host's HF token until that account
|
||||||
netavark DNATs *published* ports only: from a bridge container
|
accepts the gate. [[Model Swap and Rollback]]
|
||||||
`192.168.0.143:<published>` answers while `192.168.0.143:<host bind>` and the
|
- **A scout verdict is a lead, never a finding.** In the 2026-10-19 vault de-dup campaign six
|
||||||
gateway `10.89.1.1` are both refused. Use `host.containers.internal`
|
read-only scouts worked the same pages: one classified era pages as stale *because* they
|
||||||
(169.254.1.2). This is what 502'd `hermes.franksdank.cloud` until the Caddyfile
|
document retired payloads (that is an era page's content), one reported files that do not exist,
|
||||||
upstream was changed on 2026-09-16 (its vhost comment had claimed the LAN IP
|
one reported duplications that grep disproved (`457.9 t/s` and the ping-RTT table are solo-held
|
||||||
was open from that bridge — it never was).
|
by `Ethernet Bandwidth Math`), one cited a line number past the end of the file it judged, and
|
||||||
- **A `*.gguf` filter misses mradermacher's payloads: large quants ship as
|
one drafted an Eras table with invented windows, payloads and rollback paths. Re-verify every
|
||||||
`.partNofM` fragments.** `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF`
|
claimed duplicate, every claimed staleness and every quoted holder against the file and today's
|
||||||
carries Q4_K_M (37.00 + 37.00 + 36.97 = 110.97 GiB) and Q4_K_S (35.00 + 35.00
|
fleet state before editing; a scout yield buys no evidence.
|
||||||
+ 34.09 = 104.09 GiB) as parts, while the smaller rungs (Q2_K, Q3_K_*, IQ3_*,
|
|
||||||
IQ4_XS) are single files. Fetch the parts, verify each against the Hub's LFS
|
|
||||||
sha256, concatenate in numeric order — that is how the deployed 111 GiB file
|
|
||||||
was assembled. Listing the Hub tree and filtering `endswith(".gguf")` makes
|
|
||||||
Q4_K_S look like it does not exist.
|
|
||||||
- **A fetch script must `-f` and verify, or it reports success on an access
|
|
||||||
error.** `curl -o FILE URL` exits 0 on HTTP 401/403 and writes the error body
|
|
||||||
as the "artifact": a gated repo yielded three 153-byte shards with identical
|
|
||||||
sha256 values and a cheerful `DOWNLOAD_DONE` in 20 seconds for 104 GiB. Use
|
|
||||||
`-f`/`--fail-with-body`, compare each file's byte count to the Hub's declared
|
|
||||||
size, and check the LFS sha256 — `scripts/download-frank-medium-gguf.sh`
|
|
||||||
already does all three; ad-hoc fetches tend not to. Gated repos (e.g.
|
|
||||||
`orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF`, same 104.09 GiB Q4_K_S)
|
|
||||||
return 403 even with the host's HF token until that account accepts the gate.
|
|
||||||
- **A scout verdict is a lead, never a finding.** In the 2026-10-19 vault de-dup
|
|
||||||
campaign, four read-only scouts worked the same pages: one classified era pages as
|
|
||||||
stale *because* they document retired payloads (that is an era page's content), one
|
|
||||||
reported files that do not exist, one reported duplications that grep disproved
|
|
||||||
(`457.9 t/s` and the ping-RTT table are solo-held by `Ethernet Bandwidth Math`),
|
|
||||||
and one cited a line number past the end of the file it judged. Re-verify every
|
|
||||||
claimed duplicate and every claimed staleness against the file and today's fleet
|
|
||||||
state before editing; a scout yield buys no evidence.
|
|
||||||
|
|
||||||
## Runbook Index
|
## Runbook Index
|
||||||
|
|
||||||
|
|||||||
Executable
+175
@@ -0,0 +1,175 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""OKF v0.1 conformance linter for the docs vault (serverAppConfigs-docs).
|
||||||
|
|
||||||
|
The vault is an Open Knowledge Format bundle (omp-docs/references/okf-spec.md)
|
||||||
|
plus the maintenance discipline in okf-maintenance.md, plus this repo's own
|
||||||
|
index law (AGENTS.md -> *Documentation Policy*): an index is a locator, one row,
|
||||||
|
one clause, no numbers. Enforced here so a violation cannot be merged:
|
||||||
|
|
||||||
|
FRONTMATTER every non-reserved page has a parseable frontmatter block with a
|
||||||
|
non-empty, non-generic `type` (the spec's only hard rule)
|
||||||
|
DESCRIPTION every concept has a description that is not the title, is not a
|
||||||
|
truncation artifact (ends in an ellipsis), carries no digits and
|
||||||
|
fits the one-clause budget (<= 120 chars)
|
||||||
|
TIMESTAMP every concept has an ISO `timestamp`
|
||||||
|
INDEX-FM an index.md carries no frontmatter (it is a listing, not a concept)
|
||||||
|
INDEX-COVER every index.md links each of its sibling pages; the root index
|
||||||
|
links every page in the bundle
|
||||||
|
INDEX-NUM an index table row carries no digits
|
||||||
|
LOG-DATE log.md date headings are `## YYYY-MM-DD`, newest first
|
||||||
|
LENGTH no page exceeds 500 lines (under-split); body under 15 lines is
|
||||||
|
flagged as an over-split stub
|
||||||
|
|
||||||
|
Exit status is 1 when anything fails, so it gates a test.
|
||||||
|
"""
|
||||||
|
import datetime
|
||||||
|
import pathlib
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
|
||||||
|
RESERVED = {"index.md", "log.md"}
|
||||||
|
GENERIC_TYPES = {"file", "document", "page", "note"}
|
||||||
|
DESC_MAX = 120
|
||||||
|
BODY_MIN = 15
|
||||||
|
LINE_MAX = 500
|
||||||
|
|
||||||
|
|
||||||
|
def split_frontmatter(text):
|
||||||
|
"""Return (frontmatter_raw_or_None, body)."""
|
||||||
|
if not text.startswith("---"):
|
||||||
|
return None, text
|
||||||
|
end = text.find("\n---", 3)
|
||||||
|
if end == -1:
|
||||||
|
return None, text
|
||||||
|
return text[3:end].strip("\n"), text[end + 4:]
|
||||||
|
|
||||||
|
|
||||||
|
def field(raw, key):
|
||||||
|
if raw is None:
|
||||||
|
return None
|
||||||
|
m = re.search(rf"^{key}:\s*(.*)$", raw, re.M)
|
||||||
|
return m.group(1).strip() if m else None
|
||||||
|
|
||||||
|
|
||||||
|
def is_iso_date(value):
|
||||||
|
v = value.strip().strip("\"'").split("+")[0].split("T")[0]
|
||||||
|
try:
|
||||||
|
datetime.date.fromisoformat(v)
|
||||||
|
return True
|
||||||
|
except ValueError:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def vault_pages(vault):
|
||||||
|
return [p for p in sorted(vault.rglob("*.md"))
|
||||||
|
if ".AGENTTRASH" not in str(p) and ".obsidian" not in str(p)]
|
||||||
|
|
||||||
|
|
||||||
|
def concept_pages(vault):
|
||||||
|
return [p for p in vault_pages(vault) if p.name not in RESERVED]
|
||||||
|
|
||||||
|
|
||||||
|
def check_concepts(vault, fail):
|
||||||
|
for page in concept_pages(vault):
|
||||||
|
rel = str(page.relative_to(vault))
|
||||||
|
raw, body = split_frontmatter(page.read_text(errors="replace"))
|
||||||
|
if raw is None:
|
||||||
|
fail("FRONTMATTER", rel, "no frontmatter block")
|
||||||
|
continue
|
||||||
|
ty = field(raw, "type")
|
||||||
|
if not ty:
|
||||||
|
fail("FRONTMATTER", rel, "missing type")
|
||||||
|
elif ty.lower() in GENERIC_TYPES:
|
||||||
|
fail("FRONTMATTER", rel, f"generic type {ty!r}")
|
||||||
|
desc = (field(raw, "description") or "").strip().strip("\"'")
|
||||||
|
title = (field(raw, "title") or "").strip().strip("\"'")
|
||||||
|
if not desc or desc in ("-", "—"):
|
||||||
|
fail("DESCRIPTION", rel, "missing description")
|
||||||
|
elif desc == title:
|
||||||
|
fail("DESCRIPTION", rel, "description copies the title")
|
||||||
|
elif desc.endswith("…"):
|
||||||
|
fail("DESCRIPTION", rel, "description is a truncation artifact")
|
||||||
|
elif len(desc) > DESC_MAX:
|
||||||
|
fail("DESCRIPTION", rel, f"description is {len(desc)} chars (budget {DESC_MAX})")
|
||||||
|
elif re.search(r"\d", desc):
|
||||||
|
fail("DESCRIPTION", rel, "description carries numbers (index rows must not)")
|
||||||
|
ts = field(raw, "timestamp")
|
||||||
|
if not ts:
|
||||||
|
fail("TIMESTAMP", rel, "missing timestamp")
|
||||||
|
elif not is_iso_date(ts):
|
||||||
|
fail("TIMESTAMP", rel, f"unparseable timestamp {ts!r}")
|
||||||
|
lines = page.read_text(errors="replace").splitlines()
|
||||||
|
if len(lines) > LINE_MAX:
|
||||||
|
fail("LENGTH", rel, f"{len(lines)} lines (max {LINE_MAX})")
|
||||||
|
elif len([l for l in body.splitlines() if l.strip()]) < BODY_MIN:
|
||||||
|
fail("LENGTH", rel, "stub: fewer than %d body lines" % BODY_MIN)
|
||||||
|
|
||||||
|
|
||||||
|
def check_indexes(vault, fail):
|
||||||
|
root_index = vault / "index.md"
|
||||||
|
all_stems = {p.stem for p in concept_pages(vault)}
|
||||||
|
for index in sorted(vault.rglob("index.md")):
|
||||||
|
if ".AGENTTRASH" in str(index):
|
||||||
|
continue
|
||||||
|
rel = str(index.relative_to(vault))
|
||||||
|
text = index.read_text(errors="replace")
|
||||||
|
if text.startswith("---"):
|
||||||
|
fail("INDEX-FM", rel, "index carries frontmatter")
|
||||||
|
siblings = {p.stem for p in index.parent.glob("*.md") if p.name not in RESERVED}
|
||||||
|
for stem in sorted(siblings):
|
||||||
|
if f"[[{stem}]]" not in text:
|
||||||
|
fail("INDEX-COVER", rel, f"does not link sibling {stem}")
|
||||||
|
if index == root_index:
|
||||||
|
for stem in sorted(all_stems - {p.stem for p in index.parent.glob("*.md")}):
|
||||||
|
if f"[[{stem}]]" not in text:
|
||||||
|
fail("INDEX-COVER", rel, f"does not link {stem}")
|
||||||
|
for line in text.splitlines():
|
||||||
|
if line.startswith("|") and not re.match(r"^\|[\s-|]+$", line) and "Page" not in line:
|
||||||
|
if re.search(r"\d", line):
|
||||||
|
fail("INDEX-NUM", rel, f"row carries numbers: {line[:70]}")
|
||||||
|
|
||||||
|
|
||||||
|
def check_log(vault, fail):
|
||||||
|
for log in sorted(vault.rglob("log.md")):
|
||||||
|
if ".AGENTTRASH" in str(log):
|
||||||
|
continue
|
||||||
|
rel = str(log.relative_to(vault))
|
||||||
|
dates = []
|
||||||
|
for line in log.read_text(errors="replace").splitlines():
|
||||||
|
m = re.match(r"^##\s+(.+)$", line)
|
||||||
|
if not m:
|
||||||
|
continue
|
||||||
|
head = m.group(1).strip()
|
||||||
|
dm = re.match(r"^(\d{4}-\d{2}-\d{2})$", head)
|
||||||
|
if not dm:
|
||||||
|
fail("LOG-DATE", rel, f"heading {head!r} is not a bare YYYY-MM-DD")
|
||||||
|
continue
|
||||||
|
if not is_iso_date(dm.group(1)):
|
||||||
|
fail("LOG-DATE", rel, f"heading {head!r} is not a real date")
|
||||||
|
dates.append(dm.group(1))
|
||||||
|
if dates != sorted(dates, reverse=True):
|
||||||
|
bad = next(i for i in range(1, len(dates)) if dates[i] > dates[i - 1])
|
||||||
|
fail("LOG-DATE", rel, f"{dates[bad]} after {dates[bad - 1]}: not newest first")
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv):
|
||||||
|
vault = pathlib.Path(argv[1] if len(argv) > 1 else "serverAppConfigs-docs")
|
||||||
|
if not vault.is_dir():
|
||||||
|
print(f"not a directory: {vault}")
|
||||||
|
return 2
|
||||||
|
failures = []
|
||||||
|
|
||||||
|
def fail(rule, page, msg):
|
||||||
|
failures.append(f"{rule:12} {page}: {msg}")
|
||||||
|
|
||||||
|
check_concepts(vault, fail)
|
||||||
|
check_indexes(vault, fail)
|
||||||
|
check_log(vault, fail)
|
||||||
|
print(f"{len(concept_pages(vault))} concept pages, {len(failures)} OKF violations")
|
||||||
|
for line in failures:
|
||||||
|
print(" " + line)
|
||||||
|
return 1 if failures else 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main(sys.argv))
|
||||||
Executable
+135
@@ -0,0 +1,135 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Regenerate serverAppConfigs-docs/index.md from page frontmatter.
|
||||||
|
|
||||||
|
The root index is a locator (AGENTS.md -> *Documentation Policy*): one row per
|
||||||
|
page, one clause, no numbers. Hand-maintaining it is how rows go missing, and
|
||||||
|
the docs seam fails when a page is not linked from it, so it is generated:
|
||||||
|
|
||||||
|
* preamble everything above the first `## ` in the existing file is kept
|
||||||
|
verbatim (that is the curated statement of the vault's law)
|
||||||
|
* section from the page's directory, overridden by its frontmatter tags
|
||||||
|
(`endpoint` / `retired-endpoint` split the LLM pages)
|
||||||
|
* row text the page's frontmatter `description`, verbatim
|
||||||
|
|
||||||
|
A page with no description is a hard error rather than an empty cell, so the
|
||||||
|
generator cannot silently produce a silent index. Sections with no pages are
|
||||||
|
skipped. Deterministic: the same vault always yields the same bytes.
|
||||||
|
"""
|
||||||
|
import pathlib
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
|
||||||
|
RESERVED = {"index.md", "log.md"}
|
||||||
|
SECTIONS = [
|
||||||
|
("Inference endpoints", "endpoint"),
|
||||||
|
("Retired endpoints", "retired-endpoint"),
|
||||||
|
]
|
||||||
|
DIRECTORY_SECTIONS = [
|
||||||
|
("Dankclaw", "Dankclaw services"),
|
||||||
|
("Dankserver", "Dankserver services"),
|
||||||
|
("FrankBook-Pro", "FrankBook-Pro"),
|
||||||
|
("Desktop", "Workstations"),
|
||||||
|
("Infrastructure", "Cross-host infrastructure"),
|
||||||
|
("Architecture", "Repo architecture and policy"),
|
||||||
|
("Operations", "Operations and incidents"),
|
||||||
|
("Agent", "Agent / harness"),
|
||||||
|
("Research", "Research and methodology"),
|
||||||
|
]
|
||||||
|
PIPELINE_SECTION = "Pipeline Parallelism (series)"
|
||||||
|
LLM_FALLBACK = "LLM technique and hardware references"
|
||||||
|
CHANGE_SECTION = "Change history"
|
||||||
|
ORDER = [t for t, _ in SECTIONS] + [t for _, t in DIRECTORY_SECTIONS] + \
|
||||||
|
[PIPELINE_SECTION, LLM_FALLBACK, CHANGE_SECTION]
|
||||||
|
|
||||||
|
|
||||||
|
def frontmatter_fields(text):
|
||||||
|
if not text.startswith("---"):
|
||||||
|
return {}
|
||||||
|
end = text.find("\n---", 3)
|
||||||
|
if end == -1:
|
||||||
|
return {}
|
||||||
|
raw = text[3:end]
|
||||||
|
fields = {}
|
||||||
|
for key in ("description", "type"):
|
||||||
|
m = re.search(rf"^{key}:\s*(.*)$", raw, re.M)
|
||||||
|
if m:
|
||||||
|
fields[key] = m.group(1).strip().strip("\"'")
|
||||||
|
tags = []
|
||||||
|
m = re.search(r"^tags:\s*\[(.*?)\]$", raw, re.M)
|
||||||
|
if m:
|
||||||
|
tags = [t.strip().strip("\"'") for t in m.group(1).split(",") if t.strip()]
|
||||||
|
elif re.search(r"^tags:\s*$", raw, re.M):
|
||||||
|
block = re.search(r"^tags:\s*\n((?:\s+-\s+.*\n?)+)", raw, re.M)
|
||||||
|
if block:
|
||||||
|
tags = [l.strip().lstrip("-").strip().strip("\"'")
|
||||||
|
for l in block.group(1).splitlines() if l.strip()]
|
||||||
|
fields["tags"] = tags
|
||||||
|
return fields
|
||||||
|
|
||||||
|
|
||||||
|
def section_for(page, vault, fields):
|
||||||
|
rel = page.relative_to(vault)
|
||||||
|
if len(rel.parts) == 1:
|
||||||
|
return CHANGE_SECTION if page.name == "log.md" else "Uncategorised"
|
||||||
|
top = rel.parts[0]
|
||||||
|
if top == "LLM":
|
||||||
|
if PIPELINE_SECTION_DIR in rel.parts:
|
||||||
|
return PIPELINE_SECTION
|
||||||
|
for title, tag in SECTIONS:
|
||||||
|
if tag in fields.get("tags", []):
|
||||||
|
return title
|
||||||
|
return LLM_FALLBACK
|
||||||
|
for prefix, title in DIRECTORY_SECTIONS:
|
||||||
|
if top == prefix:
|
||||||
|
return title
|
||||||
|
return "Uncategorised"
|
||||||
|
|
||||||
|
|
||||||
|
PIPELINE_SECTION_DIR = "Pipeline Parallelism"
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv):
|
||||||
|
vault = pathlib.Path(argv[1] if len(argv) > 1 else "serverAppConfigs-docs")
|
||||||
|
index = vault / "index.md"
|
||||||
|
if not index.is_file():
|
||||||
|
print(f"missing {index}")
|
||||||
|
return 2
|
||||||
|
existing = index.read_text()
|
||||||
|
preamble = existing.split("\n## ", 1)[0].rstrip("\n")
|
||||||
|
|
||||||
|
pages = [p for p in vault.rglob("*.md")
|
||||||
|
if ".AGENTTRASH" not in str(p) and ".obsidian" not in str(p)
|
||||||
|
and p.name != "index.md" and p != index]
|
||||||
|
buckets, errors = {}, []
|
||||||
|
for page in pages:
|
||||||
|
fields = frontmatter_fields(page.read_text(errors="replace"))
|
||||||
|
if page.name != "log.md":
|
||||||
|
desc = fields.get("description")
|
||||||
|
if not desc:
|
||||||
|
errors.append(str(page.relative_to(vault)))
|
||||||
|
continue
|
||||||
|
else:
|
||||||
|
desc = "Chronological incident and law log for the whole bundle."
|
||||||
|
buckets.setdefault(section_for(page, vault, fields), []).append((page.stem, desc))
|
||||||
|
if errors:
|
||||||
|
print("pages with no description (fix the frontmatter, not the index):")
|
||||||
|
for e in sorted(errors):
|
||||||
|
print(" " + e)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
out = [preamble, ""]
|
||||||
|
for title in ORDER + sorted(set(buckets) - set(ORDER)):
|
||||||
|
rows = sorted(buckets.get(title, []), key=lambda r: r[0].lower())
|
||||||
|
if not rows:
|
||||||
|
continue
|
||||||
|
out += [f"## {title}", "", "| Page | Owns |", "|---|---|"]
|
||||||
|
out += [f"| [[{stem}]] | {desc} |" for stem, desc in rows]
|
||||||
|
out.append("")
|
||||||
|
out.append("<!-- generated by scripts/gen-vault-index.py; edit page frontmatter instead -->")
|
||||||
|
index.write_text("\n".join(out) + "\n")
|
||||||
|
print(f"{index}: {sum(len(v) for v in buckets.values())} rows in {len([t for t in ORDER if buckets.get(t)])} sections")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main(sys.argv))
|
||||||
@@ -98,7 +98,7 @@ re-run the same arms with the box quiet before a figure is carried anywhere else
|
|||||||
| Window | Payload | Runtime | Shape | Replaced because | Rollback |
|
| Window | Payload | Runtime | Shape | Replaced because | Rollback |
|
||||||
|---|---|---|---|---|---|
|
|---|---|---|---|---|---|
|
||||||
| 2026-10-18 13:19 → same evening (the boot `68afc6d` replaced) | the same 9B MTP file and projector | PrismML fork 10777 | 3 × 262,144 (pool 786,432), q8_0 K+V = 13,056 MiB, settled 17,956 / 6,169 MiB; 168.7 t/s decode (three runs 168.4–168.7), ~300 t/s over three streams (102.8 each), 2,877 t/s prefill at 106,418 | Frank ordered unquantized KV, and that pool cannot survive it: 24,576 MiB of f16 cache > 24,123 addressable | one env line — `CTX_SIZE=786432 PARALLEL=3 CACHE_TYPE_K/V=q8_0` — and redeploy; no payload move |
|
| 2026-10-18 13:19 → same evening (the boot `68afc6d` replaced) | the same 9B MTP file and projector | PrismML fork 10777 | 3 × 262,144 (pool 786,432), q8_0 K+V = 13,056 MiB, settled 17,956 / 6,169 MiB; 168.7 t/s decode (three runs 168.4–168.7), ~300 t/s over three streams (102.8 each), 2,877 t/s prefill at 106,418 | Frank ordered unquantized KV, and that pool cannot survive it: 24,576 MiB of f16 cache > 24,123 addressable | one env line — `CTX_SIZE=786432 PARALLEL=3 CACHE_TYPE_K/V=q8_0` — and redeploy; no payload move |
|
||||||
| 2026-10-18, 24 min (arm under the unit) | `Bonsai2-PTQ1_0/Ternary-Bonsai-2-27B-PTQ1_0.gguf` 5,946,648,928 B + hardlinked Q8 mmproj | PrismML fork 10777 | 2 × 200,704, batch/ubatch 3072/1536 | Frank scratched the 27B plan — ~48 t/s decode vs 76.6 for PQ2_0 on the same weights | `/home/dank/models/Bonsai2-PTQ1_0/` + `scripts/download-frank-zealot-bonsai2-ptq1.sh` |
|
| 2026-10-18 03:14 → 13:19 (arm under the unit, then the leftover drop-in) | `Bonsai2-PTQ1_0/Ternary-Bonsai-2-27B-PTQ1_0.gguf` 5,946,648,928 B + hardlinked Q8 mmproj | PrismML fork 10777 | arm: 2 × 200,704, batch/ubatch 3072/1536. The leftover `frank-zealot.service.d/ptq1.conf` then made the 13:16 cutover deploy boot THIS payload at 3 × 133,888 (`new slot, n_ctx = 133888` x3, journal read 2026-10-19) until the 13:19 boot | Frank scratched the 27B plan — ~48 t/s decode vs 76.6 for PQ2_0 on the same weights | `/home/dank/models/Bonsai2-PTQ1_0/` + `scripts/download-frank-zealot-bonsai2-ptq1.sh` |
|
||||||
| 2026-10-17 → 10-18 | Ternary-Bonsai-2-**27B** PQ2_0 7,206,168,928 B `3907dc16…` + Q8_0 mmproj 629,246,976 B | PrismML fork **10754** (`~/GitPackages/llama-prism/`) | 2 × 200,704 (pool 401,408), q8_0 K+V = 13,328 MiB, settled 22,664 / 1,461; 76.6 t/s decode, 109.4 aggregate over two streams, 691.7 t/s prefill at 166,027; MTP impossible — no head in the file | the 9B bills half the KV per token (8 full layers, not 16), which funded three slots at the native window and beat it on every rate | `/home/dank/models/Bonsai2-PQ2_0/` + the 10754 fork tree, `scripts/download-frank-zealot-bonsai2.sh`, revert the swap commit |
|
| 2026-10-17 → 10-18 | Ternary-Bonsai-2-**27B** PQ2_0 7,206,168,928 B `3907dc16…` + Q8_0 mmproj 629,246,976 B | PrismML fork **10754** (`~/GitPackages/llama-prism/`) | 2 × 200,704 (pool 401,408), q8_0 K+V = 13,328 MiB, settled 22,664 / 1,461; 76.6 t/s decode, 109.4 aggregate over two streams, 691.7 t/s prefill at 166,027; MTP impossible — no head in the file | the 9B bills half the KV per token (8 full layers, not 16), which funded three slots at the native window and beat it on every rate | `/home/dank/models/Bonsai2-PQ2_0/` + the 10754 fork tree, `scripts/download-frank-zealot-bonsai2.sh`, revert the swap commit |
|
||||||
| 2026-10-07 → 10-17 | Qwen3.5-9B-Uncensored-HauhauCS-Aggressive Q6_K 7,359,259,008 B `c0ba7beb…` + BF16 mmproj 921,704,448 B | llama.cpp master `5fc4f3c` CUDA sm_86 (`~/GitPackages/llama-master/build-cuda/`) | 2 × 200,704, exact f16 K+V, MTP off (no `nextn` tensors) | the ternary 27B bought a larger model class at the same window | `/home/dank/models/Qwen35-9B-HauhauCS-Q6K/` + the master build, `scripts/download-frank-zealot.sh` |
|
| 2026-10-07 → 10-17 | Qwen3.5-9B-Uncensored-HauhauCS-Aggressive Q6_K 7,359,259,008 B `c0ba7beb…` + BF16 mmproj 921,704,448 B | llama.cpp master `5fc4f3c` CUDA sm_86 (`~/GitPackages/llama-master/build-cuda/`) | 2 × 200,704, exact f16 K+V, MTP off (no `nextn` tensors) | the ternary 27B bought a larger model class at the same window | `/home/dank/models/Qwen35-9B-HauhauCS-Q6K/` + the master build, `scripts/download-frank-zealot.sh` |
|
||||||
| 2026-10-07 (same day) | Gemma4-26B-A4B-QAT Q4_K_M 16,796,015,520 B + BF16 mmproj 1,194,827,776 B + unused `gemma4-assistant` MTP 251,937,728 B | llama.cpp master CUDA | 1 × 200,704, f16 K + q8_0 V, ubatch 1024, MTP off; 21,868 / 2,257 settled, 139.9 t/s decode (the ~4 B-active MoE is at home here); MTP ON would cost ~1.9 GiB and boot at 319 MiB free | ONE slot serialized compaction; the 9B's lighter weights bought two slots and exact KV | `/home/dank/models/Gemma4-26B-A4B/` (ungated — plain `curl -fL`) |
|
| 2026-10-07 (same day) | Gemma4-26B-A4B-QAT Q4_K_M 16,796,015,520 B + BF16 mmproj 1,194,827,776 B + unused `gemma4-assistant` MTP 251,937,728 B | llama.cpp master CUDA | 1 × 200,704, f16 K + q8_0 V, ubatch 1024, MTP off; 21,868 / 2,257 settled, 139.9 t/s decode (the ~4 B-active MoE is at home here); MTP ON would cost ~1.9 GiB and boot at 319 MiB free | ONE slot serialized compaction; the 9B's lighter weights bought two slots and exact KV | `/home/dank/models/Gemma4-26B-A4B/` (ungated — plain `curl -fL`) |
|
||||||
@@ -129,10 +129,12 @@ re-run the same arms with the box quiet before a figure is carried anywhere else
|
|||||||
3 cells, and an armed depth-2 drafter triples it to 452.25), compute buffers at graph capture, the mmproj after the
|
3 cells, and an armed depth-2 drafter triples it to 452.25), compute buffers at graph capture, the mmproj after the
|
||||||
fit. On gemma4-class hybrids `attention_k_eq_v` does **not** shrink the allocation — the cache reserves the K/V pair
|
fit. On gemma4-class hybrids `attention_k_eq_v` does **not** shrink the allocation — the cache reserves the K/V pair
|
||||||
twice, 6.125 B/elem at f16 K + q8_0 V.
|
twice, 6.125 B/elem at f16 K + q8_0 V.
|
||||||
- **`--flash-attn on` is pinned, not decorative**: a quantized V exists only on the FA path and the engine falls back silently without it. llama.cpp has **no true fp8 KV dtype** — q8_0 is the 8-bit class (1.0625 B/elem, ~8.5 effective bits). `GGML_CUDA_FA_ALL_QUANTS` is vestigial and build-specific: the PrismML fork carries it ON (since 2026-09-17) while the master build had it OFF and still dispatched Q4/Q5 KV unconditionally (`ggml-cuda/fattn.cu` has no guard), so quantized KV never required it.
|
- **`--flash-attn on` is pinned, not decorative**: a quantized V exists only on the FA path and the engine falls back silently without it. llama.cpp has **no true fp8 KV dtype** — q8_0 is the 8-bit class (1.0625 B/elem, ~8.5 effective bits). `GGML_CUDA_FA_ALL_QUANTS` is **deprecated upstream and inert in both trees on this host** (no reference anywhere in `ggml/src/ggml-cuda/`, read 2026-10-19): master's `ggml/cmake/common.cmake` warns `use GGML_CUDA_FA_QUANTS=all instead`, and the successor's default — `GGML_CUDA_FA_QUANTS="q4_0-q4_0;q8_0-q8_0;f16-f16;bf16-bf16"` (`ggml/CMakeLists.txt:208`) — already compiles the quantized-KV kernels with the old flag OFF (master's `build-cuda` cache says `FA_ALL_QUANTS:BOOL=OFF` and it dispatched q8_0/q4_0 KV regardless). Prove a build's quantized-KV ability from its `CMakeCache.txt` `GGML_CUDA_FA_QUANTS` line, never from the retired flag or the build command you remember.
|
||||||
- **Prices come from boots, not formulas, and an arm belongs to the payload it was measured on**: 3 × 180,224 +
|
- **Prices come from boots, not formulas, and an arm belongs to the payload it was measured on**: 3 × 180,224 +
|
||||||
q8_0 V fits with Q5_K_M (21,684 used / 2,441 free) and does **not** fit with Q6_K inside the 2–3 GiB band. The
|
q8_0 V fits with Q5_K_M (21,684 used / 2,441 free) and does **not** fit with Q6_K inside the 2–3 GiB band.
|
||||||
mmproj worst case (~870 MiB) is printed before fitting but never billed by the engine's own fit check. Only
|
The mmproj worst case is printed before fitting (`estimated worst-case memory usage of mmproj is
|
||||||
|
873.10 MiB` on the Bonsai-2 27B boot) but never billed by the engine's own fit check: that boot
|
||||||
|
projected **21,439 MiB** and settled at **22,664**. Only
|
||||||
`tests/frank_zealot_unit.bats`'s fit gate can see a boot that would die at `cudaMalloc`, so every retune keeps it
|
`tests/frank_zealot_unit.bats`'s fit gate can see a boot that would die at `cudaMalloc`, so every retune keeps it
|
||||||
green — and the first verbosity-4 ledger plus settled `nvidia-smi` of a new dtype belongs on this page, because
|
green — and the first verbosity-4 ledger plus settled `nvidia-smi` of a new dtype belongs on this page, because
|
||||||
projections drift.
|
projections drift.
|
||||||
@@ -179,7 +181,10 @@ re-run the same arms with the box quiet before a figure is carried anywhere else
|
|||||||
`predicted_per_second`). Keep `VERBOSITY=4` — 3 is the engine default and cannot prove the KV or draft sizes,
|
`predicted_per_second`). Keep `VERBOSITY=4` — 3 is the engine default and cannot prove the KV or draft sizes,
|
||||||
while levels above 4 flood the journal.
|
while levels above 4 flood the journal.
|
||||||
- llama.cpp does **not** validate the requested model id: it answers and echoes the served `ALIAS`, so a rename breaks id-keyed consumers (the OpenWebUI `model` row, its `access_grant`, `openai.api_configs`) and nothing server-side. Alias history: `frank-small` → `frank-secondary` (2026-10-06) → `frank-zealot` (`7c77326`, 2026-10-09); the service and unit took the name 2026-10-10 (`55f87fc`).
|
- llama.cpp does **not** validate the requested model id: it answers and echoes the served `ALIAS`, so a rename breaks id-keyed consumers (the OpenWebUI `model` row, its `access_grant`, `openai.api_configs`) and nothing server-side. Alias history: `frank-small` → `frank-secondary` (2026-10-06) → `frank-zealot` (`7c77326`, 2026-10-09); the service and unit took the name 2026-10-10 (`55f87fc`).
|
||||||
- The first request after a boot is a warmup artifact (37 t/s class) and is never a measurement; never time a wall-clock aggregate on a generation that stops on its own (use `ignore_eos`).
|
- The first request after a boot is a warmup artifact, never a measurement: on the Bonsai-2 27B
|
||||||
|
boot request #1 gave 2.6 t/s prefill / 37.7 t/s decode and request #2 gave 1,258 t/s prefill /
|
||||||
|
54.7 t/s per stream (two concurrent, the 109.4 aggregate in the Eras row) — a 3x spread on one
|
||||||
|
shape. Never time a wall-clock aggregate on a generation that stops on its own (use `ignore_eos`).
|
||||||
|
|
||||||
## Files
|
## Files
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user