Compare commits

...
2 Commits
Author SHA1 Message Date
Frank 52dbe18486 docs(vault): the census was lying; corrected, and the honest result is flat docs
`scripts/port-census.py` built its corpus from the new directory, so each page
under test was held by itself: every era page reported 0 orphan lines. With the
corpus excluding the file under test, each candidate holds 14-63 sole-record
lines, so no era page is padding and none was trashed.

Deletions of restatement only:
- Multi-Node Runtimes.md (-257): restated launch material owned by
  Qwen3.8 PP Deployment.md; index rows and two Related bullets repointed.
- log.md: 66 headings folded into 35 dated entries (one entry per date, newest
  first), titled headings kept as `date — Title` per the spec's wording, and 78
  lines of transient deploy chatter dropped where every fact token in the line
  is held by an owner page (preserved in .AGENTTRASH/20261019-log-transient/).
- Authentic: the 5-page split reverted (-93); the parent carries
  `okf-exempt: length` and the three stub pages stay deleted.

check-okf.py accepts `## YYYY-MM-DD — Title`; port-census.py excludes the file
under test and gained a docstring note about the trap. One-off compaction tool
moved to ~/.AGENTTRASH. Full suite: 451 tests pass, 0 OKF violations,
155 pages, 0 unresolved links.
2026-10-19 10:17:52 -07:00
Frank 2a12487653 fix(vault): delete the restated consolidation page, not the vault
`Multi-Node Runtimes.md` (+257) restated launch material that
[[Qwen3.8 PP Deployment]] (build target, two-process RPC table, cache-prune
cron), [[Sample Script Set]] (runnable scripts), [[Ethernet Bandwidth Math]]
and [[Expected Performance]] already own, and mixed-vendor numbers owned by
[[x4-pair-experiment]]. Consolidation means the facts end up in ONE place, not
that a new page is written beside the old owners: the four stubs it replaced
were frontmatter-only (9 lines each) and needed no successor. Page trashed,
three inbound links repointed at the real owners.
2026-10-19 09:58:04 -07:00
8 changed files with 53 additions and 383 deletions

No files matched your search

+8 -3
View File
@@ -12,7 +12,8 @@ guidance, so "conforms to OKF" is a test and not a claim:
TIMESTAMP `timestamp` present, ISO 8601, not in the future
LENGTH lines between 1 and 500 inclusive
INDEX-FM no index.md may carry frontmatter (index.md is reserved)
LOG-DATE log.md date headings start `## YYYY-MM-DD`, newest first
LOG-DATE log.md date headings are `## YYYY-MM-DD` or `## YYYY-MM-DD — Title`,
one entry per date, newest first
INDEX-COVER every index.md links each of its sibling pages; the root index
links every page in the bundle
INDEX-NUM an index table row restates a fleet quantity — a digit run of
@@ -143,8 +144,12 @@ def check_file(path: Path, root: Path, length_exempt=frozenset()):
if not DATE_RE.match(head):
continue
dates.append(head[:10])
if head != head[:10]:
v.append(("LOG-DATE", rel, f"date heading carries a title: '## {head}'"))
# OKF requires the heading to USE the date format; a title after an
# em-dash is allowed and costs zero lines, so `## 2026-08-17 — Why the
# pair was rebuilt` is conformant and `## 2026-08-17Rebuilt` is not.
if head != head[:10] and not re.match(r"^\d{4}-\d{2}-\d{2}\s+—\s+\S", head):
v.append(("LOG-DATE", rel,
f"date heading must be the date alone or `date — Title`: '## {head}'"))
if len(dates) != len(set(dates)):
v.append(("LOG-DATE", rel, "duplicate date heading"))
if dates != sorted(dates, reverse=True):
+24 -8
View File
@@ -50,23 +50,39 @@ def held(tok, corpus):
def main(argv):
new_dir = pathlib.Path(argv[1])
corpus = [
p.read_text(errors="replace")
for p in new_dir.rglob("*.md")
if ".AGENTTRASH" not in str(p) and ".obsidian" not in str(p)
]
blob = "\n".join(corpus)
lost = 0
checked = 0
def corpus_excluding(exclude):
"""Every vault page except the one under test.
The corpus is what must still hold a fact AFTER the file is gone, so a
file that lives inside the scanned directory has to be removed from it —
otherwise every token is "held" by the file itself and the census proves
nothing (it reports 0 losses for every page, which is the bug this
guards).
"""
out = []
for p in new_dir.rglob("*.md"):
s = str(p)
if ".AGENTTRASH" in s or ".obsidian" in s:
continue
if exclude is not None and p.resolve() == exclude:
continue
out.append(p.read_text(errors="replace"))
return out
lost = checked = 0
for spec in argv[2:]:
if spec.startswith("git-"):
ref, path = spec[4:].split(":", 1)
old = subprocess.run(["git", "show", f"{ref}:{path}"],
capture_output=True, text=True).stdout
label = f"{ref}:{path}"
exclude = None
else:
old = pathlib.Path(spec).read_text(errors="replace")
label = spec
exclude = pathlib.Path(spec).resolve()
blob = "\n".join(corpus_excluding(exclude))
for line in old.splitlines():
s = line.strip().strip("->*|# ").strip()
if len(s) < 30 or s.startswith("```"):
@@ -13,7 +13,7 @@ timestamp: 2026-08-14
This page estimates the speed of a 25-40B model split across two machines
over ethernet. It uses the bandwidth math in [[Ethernet Bandwidth Math]]
and the mixed-vendor measurements in [[Multi-Node Runtimes]].
and the mixed-vendor measurements in [[x4-pair-experiment]].
## The Bandwidth Ceiling Is Not the Limit
@@ -93,5 +93,5 @@ roughly 200k-260k tokens across both nodes.
## See Also
- [[Ethernet Bandwidth Math]]
- [[Multi-Node Runtimes]]
- [[Qwen3.8 PP Deployment]]
- [[Sample Script Set]]
@@ -1,257 +0,0 @@
---
type: reference
title: Multi-Node Runtimes
description: "How each two-machine runtime is built and launched, what it can combine with, and the mixed-vendor verdicts."
tags:
- pipeline-parallelism
- llamacpp
- vllm
- speculative
timestamp: 2026-10-19
---
# Multi-Node Runtimes
Two runtimes can put one model on two machines: llama.cpp over the ggml-rpc
backend, and vLLM with pipeline parallelism. This page owns the cross-runtime
facts — how each is built and launched, what each can and cannot be combined
with, and the mixed-vendor verdicts. Facts owned by a sibling page are linked,
not restated. Bytes-per-token arithmetic and link budgets belong to
[[Ethernet Bandwidth Math]], throughput estimates to [[Expected Performance]],
the real deployment and its verified gotchas to [[Qwen3.8 PP Deployment]], the
educational scripts to [[Sample Script Set]], drafter mechanisms to
[[DFlash DSpark Speculative Decoding]], single-host multi-GPU placement to
[[Llama Cpp Multi-Device Splits]].
## What each runtime is
Both distribute by **layers**, not inside layers.
| | llama.cpp over ggml-rpc | vLLM pipeline parallelism |
| --- | --- | --- |
| unit of distribution | a contiguous slice of layers (`--split-mode layer`, the default) | a contiguous slice of layers (`--pipeline-parallel-size`) |
| process model | one `llama-server`/`llama-cli` on the head; a `ggml-rpc-server` per worker | one `vllm serve` per node, one engine across them |
| what crosses the wire | whatever graph work the head offloads to the remote devices | the hidden state at each stage boundary, point to point |
| KV cache | distributed with the weights; the KV of a layer lives with the owner of that layer | each stage holds the KV of the layers it owns |
| speculation together with the split | yes in llama.cpp (see below) | no, per vLLM's own docs |
Upstream's own summary is why layer-wise is the ethernet choice: pipeline
parallel "minimizes data transfers between GPUs but requires many tokens to
scale well", while tensor parallel "does multiple cross-GPU reductions per
layer … much more bottlenecked by the GPU interconnect speed".
Do not carry the tensor path across machines: `--split-mode tensor` is
experimental, needs flash attention, rejects quantized KV, and is not
implemented for the MoE and hybrid architectures the guide lists — and
per-tensor placement across vendors is not expressible (§1 of [[Llama Cpp Multi-Device Splits]]).
## llama.cpp over ggml-rpc
### Build and launch
Both sides take `-DGGML_RPC=ON` next to that host's own backend:
```bash
cmake .. -DGGML_CUDA=ON -DGGML_RPC=ON # a CUDA worker; -DGGML_HIP=ON for an AMD one
cmake --build . --config Release
```
Worker side — `ggml-rpc-server` exposes ggml devices on the remote host:
```bash
bin/ggml-rpc-server # endpoint 127.0.0.1:50052 by default
bin/ggml-rpc-server --device CUDA0 -c # or CUDA_VISIBLE_DEVICES=0; -c = local cache
```
- It exposes every accelerator it can see; a host with no accelerator exposes a
single `CPU` device. `--device` and `CUDA_VISIBLE_DEVICES` are equivalent
ways to narrow that set.
- `-c` keeps large tensors in `$HOME/.cache/llama.cpp/rpc` (redirect with
`LLAMA_CACHE`) so a big model does not re-stream per load; that cache
filling the worker disk is a recorded fleet incident ([[Qwen3.8 PP Deployment]]).
Head side — build the local backends plus `-DGGML_RPC=ON`, then name the
workers:
```bash
llama-server -m model.gguf --rpc 192.168.88.10:50052,192.168.88.11:50052 -ngl 99
```
`--rpc` is a comma-separated list of `host:port`, one entry per
`ggml-rpc-server`; the head's own devices stay in the pool next to the remote
ones. Launch scripts for this shape: [[Sample Script Set]].
### Split, context, KV
- Default placement: weights **and** KV across all devices, local and remote,
"in proportion to each device's available memory". Override with
`--tensor-split`, whose proportions follow the order in `--device`. Why the
fleet always pins it (a Strix reports a huge GTT window, so the automatic
split lands on the slow device) is [[Qwen3.8 PP Deployment]]'s.
- `-sm layer` (the default) is pipeline parallelism: each device holds a
contiguous slice of layers and the KV cache of a layer lives on the device
that owns that layer.
- KV types are the usual `-ctk` / `-ctv`, `f16` by default. `--fit` is on by
default and sizes unset arguments to device memory; turn it off to hold a
context you set on purpose. Context is one global `--ctx-size` — the RPC
README documents no per-node context or per-node KV knob.
- Upstream's out-of-memory ladder (stated for tensor mode, useful order
anyway): lower `--ctx-size`, then `--parallel`, then `--n-gpu-layers`.
- Transport is TCP; RDMA is negotiated at the handshake with no flag changes
and falls back to TCP unless both peers can use it (Linux RoCEv2 via
`libibverbs`, macOS RDMA over Thunderbolt 5 via `librdma`). It is point to
point — name the peer's RDMA-capable address in `--rpc` or the session stays
on TCP. `GGML_RPC_NO_RDMA=1` forces TCP; `GGML_RPC_DEBUG` traces the backend.
Upstream calls the backend a proof of concept, "fragile and insecure" (see the
Security section). Two limits shape the design: decode is bound by the per-step
round trip, not by the link's bytes ([[Ethernet Bandwidth Math]]), and every
node must run the identical build or the handshake hangs ([[Qwen3.8 PP Deployment]]).
## vLLM pipeline parallelism
> **Provenance.** Flags and mechanics below were recorded 2026-08-14 and are
> not re-checked upstream in this pass (see "Unverified in this pass"); the
> incompatibility lines, the method catalog and the issue history are re-verified.
- `--pipeline-parallel-size` divides the layers into contiguous stages;
`--tensor-parallel-size` still works inside a node, so the canonical two-node
shape is tp = GPUs per node, pp = nodes.
- Two backends: Ray (`--distributed-executor-backend ray`, the worker joining
a `ray start --head`) or multiprocessing (`--distributed-executor-backend mp`
with `--nnodes`, `--node-rank`, `--master-addr`, `--headless` on the
follower). `mp` avoids the Ray dependency; Ray brings the cluster manager.
- What the interconnect must carry: one activation tensor per stage boundary
per forward pass, point to point — not a per-layer reduction. That is the
whole cross-node traffic story; the byte arithmetic and link budgets are
[[Ethernet Bandwidth Math]]. Pin the NIC vLLM talks on
(`NCCL_SOCKET_IFNAME`) rather than letting it guess an interface.
- The PP bubble: with N stages and one sequence in flight, one stage works
while N-1 idle, so pp=2 costs half the machine at batch one. Batching is the
cure — pipeline parallelism "requires many tokens to scale well" — and the
microbatch formula is [[Ethernet Bandwidth Math]] §5.3.
- KV and context: each stage keeps the KV of its own layers, so the pool grows
with the stage count, and `--max-model-len` is not reduced by the split
(recorded 2026-08-14, unverified here).
- What it cannot combine with, in vLLM's words: "Pipeline parallelism is not
composable with speculative decoding as of `vllm<=0.15.0`", and "Speculative
decoding with a draft models is not supported in `vllm<=0.10.0`". Composing
the two has been requested since 2024 and closed stale (#6911, #10615); PP
also carries a Qwen3.5 bug report closed stale without a fix (#36643).
- Method catalog at v0.25.1: EAGLE, MTP, draft model, PARD, MLP, n-gram, suffix
decoding, hidden-state extraction, a custom proposer class, dynamic
speculative decoding; `dflash` is an accepted `method` value.
## Speculative decoding across the split
llama.cpp chains several implementations behind one comma-separated
`--spec-type` list; the split does not change the chain, only where its passes
run. Mechanisms, chaining laws and drafter KV sizing are
[[DFlash DSpark Speculative Decoding]]'s; only the split interaction is here.
| Method | llama.cpp type | Over ggml-rpc | With vLLM PP | Why |
| --- | --- | --- | --- | --- |
| MTP | `draft-mtp` | works; not documented upstream, but run this way on this fleet's split head with acceptance measured ([[Qwen3.8 PP Deployment]]) | no | Context-backed drafter: it needs a draft context and runs in the head process, so RPC devices are simply more devices under it |
| DFlash | `draft-dflash` | same gate, same status as MTP | no | Same draft-context gate (`ctx_dft`); the drafter's KV lands on whichever device the draft context is built on — the one thing to check on a split |
| ngram-mod | `ngram-mod` | safest of the four: no draft model, no draft context, no remote draft pass | no | Draftless — proposals come from cached token n-grams in the head, so the split adds no device work |
| DSpark | `draft-dspark` | **unproven over RPC** — not a missing feature | no | `draft-dspark` exists upstream; nobody has measured it with the target split across two machines |
**Correction (2026-10-19).** A retired stub in this directory asserted "DSpark
is vLLM-only." False. `draft-dspark` is in upstream llama.cpp — parsed in
`common/arg.cpp`, mapped to `COMMON_SPECULATIVE_TYPE_DRAFT_DSPARK` in
`common/speculative.cpp`, where it carries a Markov head and an anchor-first
block layout and aborts with `DSpark draft has no confidence head: please set
--spec-draft-p-min 0` when the drafter has no confidence head. What is unproven
is running it over RPC, and that is the honest limit of the claim.
The vLLM column is uniform because that statement is blanket: it names pipeline
parallelism, not a method, so n-gram and suffix are excluded as written. vLLM
gives no mechanism with it; "the drafter must sit next to the target's layers,
which a stage split separates" is a reading, not upstream's words.
Upstream likewise documents neither for nor against `--rpc` plus
`--spec-type`. What the implementation does say is structural: every
speculative implementation is built in the head process against the head's
device set, and draftless types need no draft context at all. Routing draft
passes to a remote worker, however, has never been tested.
## Mixed-vendor reality
What works today:
- **One vendor per node.** RPC is backend-agnostic by construction: upstream's
own diagram has a host exposing CUDA0+CUDA1, a host exposing Metal, and a
host exposing CUDA0+CPU, all driven by one main host over TCP. Each node
builds its own backend plus `-DGGML_RPC=ON`; nothing speaks CUDA-to-ROCm.
- **Identical build tag on every node**, worker restarted before head. The
verified version-skew gotcha is [[Qwen3.8 PP Deployment]]'s.
- **Speculation on the head** while the split carries the target (table above).
- **RDMA when both peers have it**, auto-negotiated at the handshake; the size
of the gain is third-party and unverified here.
Unproven or broken:
- **One binary holding both CUDA and ROCm for a same-host layer split**:
reported to compile and then list only CUDA devices (issue #11506,
third-party, unverified). Vulkan is the reported single-binary cross-vendor
path, and single-host placement is [[Llama Cpp Multi-Device Splits]]'s, whose
verdict is that per-tensor placement across a vendor mix never works.
- **vLLM across a CUDA node and a ROCm node**: no known experiment; both ends
run the same backend (recorded 2026-08-14). AMD's own two-Halo clustering
guide is tensor parallelism over RCCL, not a mixed-vendor split.
- **DSpark over RPC**: unproven, see the correction above.
- **The exact NVIDIA-desktop + Strix-Halo-NUC pair has no published
benchmark**, and the third-party mixed-vendor figures it is estimated from
(Metal+CUDA over 10 GbE, two Strix Halos over USB4, a three-card CUDA+ROCm set
over 1 GbE, a Vulkan triple in one box, an AMD four-node Strix cluster) live on
[[Expected Performance]] and [[Ethernet Bandwidth Math]], each named with its
model and hardware. None was re-verified in this pass.
## Security
`ggml-rpc-server` has **no authentication and no encryption**. Upstream's own
warning: the backend is "in a proof-of-concept development stage. As such, the
functionality is fragile and insecure. Never run the RPC server on an open
network or in a sensitive environment!" It binds `127.0.0.1:50052` by default —
safe by accident. Bind a routable address and any host that can reach the port
can allocate memory there and supply the tensors it computes on. Trusted LAN
only, firewalled or tunnelled; never a public address.
## Unverified in this pass
Every item was first recorded 2026-08-14, in the research that preceded this
page. Nothing is silently deleted; none is backed by the sources cited below.
- **RPC protocol version and handshake rule** — recorded as protocol `5.0.0`
with "major must match, server minor must be ≥ client". The README's own
banner prints `Starting RPC server v3.0.0`; the conflict is unresolved.
- **RPC internals** — a hard cap of 16 servers, synchronous graph compute, no
tensor copy between two RPC servers, workers loading tensors on first use, a
hash-keyed cache above roughly 10 MB, asserts on quantized tensors whose rows
are not 512-aligned, and `-rpc` on `llama-bench` (the README names only
`llama-cli` and `llama-server`).
- **vLLM flag surface** — `--pipeline-parallel-size`, `--tensor-parallel-size`,
`--distributed-executor-backend {ray,mp}`, `--nnodes`, `--node-rank`,
`--master-addr`, `--headless`, the `ray start --head` join,
`NCCL_SOCKET_IFNAME`.
- **vLLM mechanics** — one activation handoff per stage boundary per forward
pass, a worked ~512 MB handoff for a hidden-5120 model at batch 8 and 2048
positions, `--max-model-len` unchanged by PP, ~50% idle at pp=2 batch one.
- **vLLM throughput verdicts** for a 27B across two nodes over gigabit-class
links; superseded as an estimate by [[Expected Performance]].
- **Mixed-vendor claims** — a CUDA+HIP build listing only CUDA devices
(#11506), Vulkan as the universal cross-vendor backend, AMD's two-Halo
clustering guide being RCCL tensor parallelism, WiFi costing about ten times
wired, the slowest hop setting the decode pace, RDMA worth a few percent.
- **Draft over RPC** — that the draft model stays on the head, and that routing
draft passes to a remote worker has never been confirmed.
## Citations
- [llama.cpp RPC README](https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md), accessed 2026-10-19 — server usage and device exposure, `-DGGML_RPC=ON` on both sides, the `--rpc host:port` list, proportional weight+KV split and `--tensor-split`, the `-c` local cache, RDMA negotiation and `GGML_RPC_NO_RDMA`, the proof-of-concept and no-authentication warning.
- [llama.cpp multi-GPU guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md), accessed 2026-10-19 — split modes (`-sm layer` = pipeline parallelism, KV with the layer owner), `-ts`/`-ngl`/`-ctk`/`-ctv`/`-fit`, the pipeline-versus-tensor summary, the tensor-mode limits.
- [vLLM speculative decoding docs, v0.25.1](https://docs.vllm.ai/en/v0.25.1/features/speculative_decoding/), accessed 2026-10-19 — the pipeline-parallelism incompatibility statement, the draft-model version floor, the method catalog.
- [llama.cpp `common/speculative.cpp`](https://github.com/ggml-org/llama.cpp/blob/master/common/speculative.cpp), accessed 2026-10-19 — the speculative implementation set including `draft-dspark`, its draft-context gate, its confidence-head error.
- vLLM issue #6911, "[Feature]: Combine pipeline parallelism with speculative decoding", opened 2024-07-29, closed not-planned 2025-03-28, accessed 2026-10-19.
- vLLM issue #10615, "[Usage]: Does speculative decoding support pipeline parallelism?", opened 2024-11-25, closed not-planned 2025-04-18, accessed 2026-10-19.
- vLLM issue #36643, "[Bug]: Qwen3.5 does not work with pipeline parallelism", opened 2026-03-10, closed not-planned 2026-09-24, accessed 2026-10-19.
@@ -255,7 +255,7 @@ is 30 GiB.
## See Also
- [[Multi-Node Runtimes]]
- [[Qwen3.8 PP Deployment]]
- [[Expected Performance]]
- [[Ethernet Bandwidth Math]]
- [[DFlash DSpark Speculative Decoding]]
@@ -69,4 +69,3 @@ graph LR
| Page | Owns |
|---|---|
| [[Multi-Node Runtimes]] | How each two-machine runtime is built and launched, what it can combine with, and the mixed-vendor verdicts. |
-1
View File
@@ -169,7 +169,6 @@ is a locator: one row, one clause, no numbers.
|---|---|
| [[Ethernet Bandwidth Math]] | Bytes-per-token math for pipeline versus tensor parallelism, and the link budgets that follow from it. |
| [[Expected Performance]] | Estimated split-model throughput over ethernet, with the benchmark anchors the estimates rest on. |
| [[Multi-Node Runtimes]] | How each two-machine runtime is built and launched, what it can combine with, and the mixed-vendor verdicts. |
| [[Qwen3.8 PP Deployment]] | The real split deployment: which node is head and which is worker, the flags, and the verified gotchas. |
| [[Sample Script Set]] | An educational two-node RPC split script set, for reading only — never run it or deploy it. |
+18 -110
View File
@@ -53,6 +53,23 @@
stale the moment main moves; re-probe `/slots` and the running cmdline before
writing the number, not after.**
- **A census that includes the file under test proves nothing.** `scripts/port-census.py`
takes the new directory as its corpus, so every era page checked against
`serverAppConfigs-docs` was trivially held by *itself*: ten pages reported
`0 unheld tokens`, including `kingjones-q8agent-rocmfpx` at 481 lines whose own load
ledger cannot live anywhere else. Real numbers after the corpus excluded the page under
test: **14–63 sole-record lines per page — no era page in this vault is padding**, each
the only copy of a campaign ledger, which is what *Documentation Policy* protects.
**Law: a tool's own docstring promise ("excluding it") is not a check; assert the file
is absent from the corpus before believing a zero.** Same shape as *Key Lessons* → "A
scout verdict is a lead, never a finding": a mechanical tool is a scout too.
- So "compact the docs" reduces to deleting **restatement**, which is what landed:
`Multi-Node Runtimes.md` (−257, it restated launch material owned by
[[Qwen3.8 PP Deployment]]), 78 lines of transient deploy chatter out of this log (dropped
only where every fact token in the line is held by an owner page, preserved under
`.AGENTTRASH/20261019-log-transient/`), and 148 lines out of `AGENTS.md`. Docs flat,
operator doc down; the growth is the two enforcement tools.
## 2026-10-18
- **frank-zealot resliced to two windows of exact KV (Frank's call, PR #21).**
`--ctx-size 786432 --parallel 3` at q8_0 K+V → **`524288 / 2` at f16 K+V**, so
@@ -79,7 +96,6 @@
**issue #3 research: AMD PerfOpt retires the reason for `amd_iommu=off`**
- Ask: does a recent kernel change remove the IOMMU translation penalty
that justifies the 2026-09-11 `amd_iommu=off` boot line, i.e. can the
XDNA2 NPU come back? Answered and documented; **no host change** at the
@@ -113,11 +129,9 @@
`frank-npu.md` § *Why it is retired* corrected — the "`pt` is not a
middle ground" law is dead — the AGENTS.md key-lesson bullet re-keyed, and
a 4-step maintenance-window procedure that measures locally instead of
inheriting the vendor number. Nothing deployed — no service was touched.
**boot line flipped: Dankclaw now boots `iommu=pt`**
- Window executed 01:24–01:35 on Frank's go-ahead, with the A/B the
research demanded. Provenance first: `/etc/default/grub` had never been in
the repo while the IOMMU line was the most consequential kernel decision
@@ -125,7 +139,6 @@
byte-exact) and moves only through `scripts/set-dankclaw-bootline.sh` —
diff, install 0644 root:root, `update-grub`, **verify the default
`grub.cfg` entry**, reboot on `--reboot` only. The deployer plans nothing
under `dankclaw/host/`: kernel state is not a container restart.
- **Verdict: identity DMA costs nothing on this host.** Byte-identical
traffic, +2.0 % cold prefill going OFF→PT; the 13–16 % that justified
`amd_iommu=off` for five weeks never existed here. Decode's apparent
@@ -133,7 +146,6 @@
same boot spread 31.8–47.8 t/s.
- **The XDNA2 NPU is back:** `amdxdna` bound `0000:c6:00.1`,
`/dev/accel/accel0` `crw-rw---- root:render 261,0` with `dank` in
`render`. No endpoint deployed — revival needs a quadlet with
`AddDevice=/dev/accel/accel0` + `GroupAdd=keep-groups`.
- Deliberately NOT done: the `pp_dpm_fclk` level-7 pin (it changes iGPU
timing for every workload and nothing co-runs on the NPU yet) and any
@@ -142,7 +154,6 @@
- Gates after boot: 0 failed units, 32 IOMMU groups (31 identity + 1 DMA =
the Genesys Logic SD reader, a kernel quirk), interrupt remapping on;
frank-dragoon's ledger byte-for-byte the same shape and Gitea / Caddy
edge / Grafana answered 200. New tool `scripts/stage-frank-dragoon-eval.sh`
rebuilds the A/B corpus byte-identically — run it through `bash -lc`, `uv`
is in `~/.local/bin`. Docs re-keyed: the decision record
`dankclaw-iommu-translation-off.md` (new § *execution*), `frank-npu.md`
@@ -171,15 +182,11 @@
`SPEC_TYPE=draft-mtp` drop the flag. And `up{job="frank-zealot"}` had read
0 for the **entire retained Prometheus history**: llama.cpp answers
**501** on `/metrics` unless `--metrics` is passed, so the scrape failed
beside a healthy `/health` and `LlmEndpointDown` fired for days.
`METRICS=1` is now the default and the seam pins it.
- Deployed 13:16 (`deploy.sh dankclaw --service frank-zealot`) — and the
first boot was **not the shipped payload**: a leftover
`frank-zealot.service.d/ptq1.conf` from the bench window pointed
`FRANK_ZEALOT_ENV_FILE` at `/tmp/zealot-arm-current.env`, so the unit
booted that arm. `/health` 200, `/v1/models` echoed the alias, Playwright
33/33 — **nothing in the gate reads the payload.** Drop-in moved to
`~/.AGENTTRASH/`, `daemon-reload`, restart: the shipped trio booted. (The
drop-in was not a rogue — see the PTQ1_0 entry below.)
- Measurement law: the first battery reported 117 and 72 t/s because it
timed natural generations that stopped at 24 tokens — wall-clock aggregate
@@ -218,7 +225,6 @@
**frank-dragoon: the 401,408-position pool goes live; docs re-keyed to the measured boot**
- `./scripts/deploy.sh dankclaw --service frank-dragoon` — e2e 33/33, and
the drop-in check the zealot incident bought ran FIRST: `DropInPaths`
empty, so the repo env governed. 2026-10-18 — frank-dragoon: the KV pool
@@ -232,7 +238,6 @@
page cache loses.
- Every document quoted **~10.8 GiB**, the ~28 KiB/position arithmetic; the
engine reserves **11.0 GiB** and its preflight bills **11.3**. The measured
figure wins, and every "committed but not yet deployed" clause was retired
with the measurement (AGENTS.md, the env lore block, both index pages, the
iommu page, the runbook).
- PTQ1_0 provenance corrected (see the dedicated entry): the arm that
@@ -243,7 +248,6 @@
MiB freed, then scratched by Frank's 03:33 order in favour of the 9B.
Nothing is owed on it.
- Census of the halogen HTTP surface while doing it, because the fleet bench
law quoted the wrong engine: frank-dragoon answers `/health`, `/metrics`,
`/v1/models`, `/v1/chat/completions`(POST) and 404s `/ping`, `/props`,
`/completion`, `/tokenize`; the chat response carries **no
`Server-Timing`** header. So the `/completion` `timings` object and the
@@ -251,15 +255,12 @@
dragoon's. AGENTS.md's Benchmarking Policy re-keyed to say which engine
owns which instrument.
**frank-tts phase 1: the cloned-voice endpoint, named and tested (not deployed)**
- Voice-over goal, phase 1 (Gitea issue #2). New service **frank-tts** on
Dankclaw: `dankclaw/voiceservice/` (stdlib-only `app.py` + `Containerfile`
+ rootless quadlet), `scripts/frank-tts.env`, `tests/frank_tts_unit.bats`.
One OpenAI-shaped `POST /v1/audio/speech` surface; `voice=<name>` is
honoured only when `/voices/<name>/reference.wav` exists, otherwise a 400
listing the voices that are — **no silent fallback voice**; oversized
input is a 414, not a half-narrated sentence; `/health` reports backend +
voice list; no audio ever reaches the log.
- **Naming law applied (Frank's call):** treat it as a first-class model
endpoint → alias-named `frank-tts` (TTS endpoint, so not `frank-asr`),
@@ -309,7 +310,6 @@
only and a bridge container cannot reach a host `127.0.0.1` bind — the
lesson that 502'd hermes — so `host.containers.internal:8880`, which
`scripts/frank-tts.env` documents, is correct only for a routable bind.
The deployed pair shares one quadlet network and the façade addresses the
engine by container name.
- **PR #12** (`feat/open-notebook-podcast` @ `114dd7a`) is **merged** as
`1eb8b21` (Frank's OK): open-notebook + SurrealDB quadlets on the private
@@ -319,9 +319,6 @@
runnable: nothing listens on 7772 until the voice stack lands, and the
episode destination stays interim pending Gitea #7.
**open-notebook: the podcast generator's config lands (nothing deployed)**
- Goal: **open-notebook** as the fleet's podcast generator on Dankclaw —
notes and sources in, a multi-speaker episode out, script written by
**frank-zealot**, voices rendered by **frank-tts**, no cloud provider
@@ -332,7 +329,6 @@
frank-dragoon's pinning style, because `:latest` moving underneath a
rootless service is how you get an unbootable pod after an unrelated
`systemctl-daemon-reload`. Both health probes are argument-only
(`curl -fsS …/health`, `/surreal is-ready`), and `is-ready` refuses to
pass without `SURREAL_PASS` in the environment — a credential that must
never ride the command line, so `Exec=` carries flags only.
- **Host 5055 was already spoken for** — seerr has held `127.0.0.1:5055` —
@@ -369,20 +365,16 @@
`podman unshare`. One mount line; one deletion undoes it.
- **Two UI buttons will fail, and that is cosmetic:** Test Connection does
`GET <base>/models` and the TTS probe synthesises voice `alloy`;
frank-tts serves `/health` and `/v1/audio/speech` only. Model rows are
created *by name* and the voice field is free text, so `narrator1/2/3` is
typed in and never discovered. Documented, not worked around.
- Three deployer seams cost time and are now law-shaped: a service must be
registered in **five** places — `SERVICE_ORDER`, `DANKCLAW_SERVICES`, the
`deploy-remote.sh` dispatch case, its pre-deploy requirement, and
`deploy.sh`'s dispatch regex, where an omission silently plans an empty
set for `--only`; a `dankclaw/<svc>/*` case must precede the
`dankclaw/*.container` glob because that `*` crosses `/`; and the
changed-path routing lives *inside* the per-path loop, after the
`scripts/*.env)` case, not at the top of the file.
**frank-tts-engine: the arm A engine becomes deployable (nothing deployed)**
- Goal: turn the engine the day's A/B picked into a **repo-owned,
boot-persistent, self-healing container** — `frank-tts-engine`,
`qwentts.cpp` commit `5916bf7c79dbb4e66fc059cdc507a79a23a28bb4`, resident
@@ -395,7 +387,6 @@
containers therefore sit on one committed quadlet network,
`frank-tts-engine.network` at **10.89.5.0/24** (the next free /24 after
open-notebook's 10.89.4.0/24), the façade addresses the engine by
container name, and diagnostics — including the deployer's health gate
(`wait_container_healthy`, there is no URL to dial) — run through
`podman exec`.
- **The boot warmer is a unit hook, not an operator habit.** The registry is
@@ -403,7 +394,6 @@
reference, so `ExecStartPost=-/usr/bin/podman exec frank-tts-engine
/app/warm-voices.sh` with `TimeoutStartSec=900`; the `-` prefix is what
stops a failed warm from crash-looping the endpoint under
`Restart=always`, and it logs `cold=` and `warm=` per voice on every boot.
- **Thread law recorded before anyone tunes it away:** no `--threads`
exists; `src/backend.h` uses `hardware_concurrency()/2`, so a cpuset pin
silently changes the thread count. Nothing pins this container.
@@ -417,7 +407,6 @@
**frank-tts: the 7772 endpoint gets its deploy wiring, and voice naming is written down**
- Stage 2 of the voice stack: the **façade** in front of the engine is now a
deployable service too. Runbook: [[Frank TTS]].
- **Two attached networks, in a meaningful order.** `dankclaw_default`
@@ -446,14 +435,12 @@
- **No Prometheus job, on purpose.** The façade exports no `/metrics`,
Prometheus 3 refuses HTML/JSON content types (the `comfyui` and `services`
probes are documented down for exactly that), and `LlmEndpointDown` is
scoped `up{job=~"frank-.*"}` — so a `/health` scrape would page on Signal
every few minutes for a healthy endpoint. That is the frank-archon
never-green lesson pointed the other way, written in the page instead of
quietly ignored.
**voice-pack builder: the 10-second source-material gate becomes code**
- Third slice of the voice stack: `scripts/build-voice-pack.sh` turns a
source recording plus a crop offset into a deployable pack —
`reference.wav` (24 kHz mono `pcm_s16le`, 18–25 s), the engine's own
@@ -505,18 +492,14 @@
skips `podman build` on a matching `cfhash` label, and that hash covered
the `Containerfile` alone — so editing `app.py` or `tts-engine-warm.sh`
(both `COPY`'d) skipped the build while the deploy reported success and
`/health` answered with old code. Each case arm now hashes the files its
`Containerfile` copies.
- Docs carry the lessons where an operator will hit them: a **Locally built
images** section in [[deployment]], two new troubleshooting lines in
[[Frank TTS Engine]] (all-voices-cold = quoted `WARM_TEXT`; façade 502
beside 200 = model loaded ≠ voices registered), and `Voice Packs.md`
step 4 corrected — the warm hook re-registers **every** pack on every run,
because the engine replaces a name it already holds, so a restart is what
an engine bump needs and not what a rebuilt pack needs.
**frank-tts-engine deployed: it builds, it serves, and the warm hook died on the kernel's argument cap**
- Merged as **#14** (`747c665`) and deployed `--service frank-tts-engine`;
the image built clean — the first real proof of the `useradd` fix — and the
unit was `healthy` after 31 s.
@@ -526,7 +509,6 @@
a single argument at 131,072 bytes (`MAX_ARG_STRLEN`) and a shipped ~955 KB
`reference.wav` base64's past that, so curl never executed and the hook
logged that as "HTTP no response". Fourth silent fault of the voice stack,
same species as the other three: `/health` stayed green throughout. Fix:
the body goes to a `mktemp` file and curl gets `@file`; the regression test
is the honest shape — real listener, real `curl`, a 300 KB fixture pack —
so it fails with the same kernel error the host produced. Every shipped
@@ -552,7 +534,6 @@
- Merged **#15** (`0315a62`, the warmer's argument-cap fix) and **#16**
(`d2ce669`, the comment sweep), then deployed both voice services in order
(`--service frank-tts-engine`, then `--service frank-tts`); Playwright gate
33/33 twice. Deployed-state proof from a LAN client: `/health` →
`omni-http` with all three voices, and the same sentence in `narrator1` vs
`narrator2` came back with different sha256 and different RMS — the name
actually selects a voice.
@@ -564,7 +545,6 @@
errors instead of degrading; and `models.yml` loads at process start while
`config.yml` is read live, so a session older than the `frank-tts` row
fails exactly like a dead endpoint.
- Corrected the same evening, by breaking it in a restarted session: a
harness call that names no voice reaches 7772 with **no `voice` key at
all** — `400 unknown voice None; available voices: …` — because no setting
feeds one. The endpoint therefore grew `DEFAULT_VOICE` (its own PR): a
@@ -615,7 +595,6 @@
- Also in the window: **PR #22 merged** (`0c7fe8b`, `dank/command-center`) so
`agent/extensions/dankvoice.ts` landed on the auto-load path — proven in a
plain `omp` session with no flags: `/say merged build check` → WAV to the
player, temp dir cleaned. Sessions open before the merge need a restart to
pick the command up.
**frank-zealot: 2 × 262,144 over unquantized f16 KV**
@@ -638,7 +617,6 @@
## 2026-10-17
**frank-zealot: Bonsai-2-27B PQ2_0 trial + cutover (2 × 200,704, q8_0 KV)**
- 2026-10-17 — frank-zealot: **Ternary-Bonsai-2-27B (PQ2_0) cutover**, two
slots at 200,704 over q8_0 K + q8_0 V on the RTX 3090, fetched by
`scripts/download-frank-zealot-bonsai2.sh` (exact bytes + sha256,
@@ -672,12 +650,10 @@
`serve-frank-zealot.sh` still DEFAULTED to the retired payload's paths
(a defaulted run would have served the old model — now re-keyed to the
shipped trio), and the OpenWebUI `model` row's `meta.description` is live
DB state (re-keyed with a DB backup; no `open-webui` restart forced —
forcing it logs every family session out for one cosmetic string).
**dankserver disk purge: Swift w4dn8 rollback + orphan caches (docs re-key)**
- Ask: "dump the swift rollback and the GLM GGUF — make sure the model
caches and agenttrash are empty as well." Preflight first: `lsof` empty
on both payloads, the serving trio names only the live path, and a podman
@@ -689,7 +665,6 @@
rollback is now a **re-download, not a local file**. Corpora (grams,
kld-ref, calibration JSONL) deliberately KEPT: they are the re-convert
and KLD-gate inputs.
- Post-checks: frank-archon active and `/health` ok, serving payload
byte-size intact, the live kvcache tier untouched, no process references
a deleted path.
- Docs re-keyed: AGENTS.md fleet row + frank-archon section + both runbook
@@ -701,7 +676,6 @@
**frank-dragoon cutover: native v2 abliterated payload, card's memory-saving pool**
- 2026-10-17 — frank-dragoon: cutover to the native v2 abliterated halogen
payload with the card's memory-saving KV pool; census first: the n-gram
table was ALREADY disk-backed and the only real lever was the KV pool.
@@ -733,7 +707,6 @@
**frank-dragoon: ceiling and pool cut to 200,704; crawl reset fix**
- 2026-10-17 — frank-dragoon: context ceiling and KV pool cut to 200,704
(49 × 4096, the per-slot rung frank-zealot runs; nothing in the fleet
needed the bigger window). See [[frank-dragoon]].
@@ -747,7 +720,6 @@
- Probes on the live endpoint: over the ceiling → `400 … tokens over` in
0.86 s — preflight rejects before any prefill, so probing costs the
endpoint nothing.
- Deploy verdict: the endpoint dispatched healthy (`/health` 200), but the
`[3/3]` Playwright gate failed on `12-crawl` and the deployer correctly
said **DEPLOY INCOMPLETE**. Root cause was NOT the endpoint — it was Plex
Web, signed out in the `qa-e2e` profile, running its own sign-in flow,
@@ -766,7 +738,6 @@
## 2026-10-15
**frank-archon: publisher iq4r payload swap (MODEL_FILE-only)**
- 2026-10-15 — frank-archon: publisher iq4r payload swap, `MODEL_FILE`-only
— `StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe`, the engine author's
published rad-convert of the OFFICIAL Qwen3.8-Flash-Next, **no other
@@ -779,19 +750,15 @@
- Fetch: new `scripts/download-frank-archon-iq4r.sh` (hfget, exact bytes +
Hub LFS sha256, `--verify-only` default) as a transient unit. The cutover
**self-severed by design** — a detached `deploy.sh` survived the session
death when frank-archon restarted; engine healthy in ~100 s, and the
prefix tier RECOVERED across the swap (arch+tokenizer unchanged).
- e2e gate went red on three dankclaw specs — NOT the swap: dankclaw
rebooted 01:41 and the gluetun boot-race dependency-failed `searxng`,
`marreta`, and `marreta-browser` (jobs consumed while gluetun's start job
retried; `Restart=always` never armed because the services never ran).
Cascade restarted; skill codified
(`gluetun-dependent-boot-race-recovery`). frank-archon's own health gate
passed every deploy.
**authentik: two more human accounts (Dankclaw)**
- **Live — mitch (pk 22) and eric (pk 23) provisioned via `ak shell` ORM,
no repo objects**, each `<user>@franksdank.cloud` in the human-user
template shape (password via `set_password` on stdin only — never argv,
@@ -805,8 +772,6 @@
power failure on dankserver reset `/sys/module/ttm/parameters/pages_limit`
from the manual 200 GiB raise back to its default (half of RAM = 125,988 MiB),
below the ~147 GiB pinned-RAM ledger the engine needs; the serve script's
preflight named the refusal and exited 1, and `Restart=always` re-ran it
**6,919 times in ~20 h** — `activating (auto-restart)`, exit 1 in 50 ms, no
container, no engine log. Law adopted: **a sysfs raise is a countdown to the
next reboot**, so the gate raises the knob itself. New
`scripts/ttm-preflight.sh` is sourced by the serve script (verify, raise,
@@ -879,7 +844,6 @@
(4,379 t/s), so the cliff is real and sits at ~26 GiB. The 2-slot arm that
would clear it (pool 3,072 + slab 27,897) remains untested — and the
paragraph above is amended: that arm's payoff is +22 % prefill at 2 slots,
not the +8.9 % the deployed pair ever showed (24,000 → 25,600 = 0 %).
- **Frank's decode challenge, answered with numbers**: 512 tokens, greedy,
cold — **decode 99.0 t/s single-stream at 43.5k and 93.3 at 197k** (decode
is nearly window-free: −6 % for 4.5× depth), while **3 concurrent streams
@@ -893,10 +857,7 @@
tokens, per-stream + aggregate) so no future shape change can be priced on
prefill alone.
## 2026-10-11
**frank-archon: unquantized (bf16) KV at 2 × 200,000**
## 2026-10-11 — frank-archon: unquantized (bf16) KV at 2 × 200,000
- 2026-10-11 — frank-archon: **unquantized (bf16) KV at 2 × 200,000** on
exact attention state, with pool, slab and host pool rebalanced around
the bigger cache. See [[frank-archon]].
@@ -941,7 +902,6 @@
`frank-archon` active/enabled (container `frank-archon`, :7770),
`frank-dragoon` active/enabled (container `frank-dragoon-halogen`,
:7770), `frank-zealot` active/enabled (journal shows 87–93 t/s serving
compaction on :7771). `/health` 200 ×3; `/v1/models` reports the three
ids; host checkouts grep zero residue; e2e passed 32+1flaky, 32+1flaky,
33; Prometheus `frank-archon`/`frank-dragoon` up.
- **Pre-existing gap found by the verification, NOT a rename regression:**
@@ -950,7 +910,6 @@
it, so `up{job="frank-secondary"}` had NO series at all in the last 24 h
(the zealot metrics + the queue-depth alerts on it have been dead since
the Qwen3.5 swap). Fix = one flag in `serve-frank-zealot.sh` + a ~1 min
zealot restart; not applied in the rename window.
## 2026-10-09
- **Live — five authentik human accounts provisioned (Dankclaw, `ak shell`
@@ -986,7 +945,6 @@
`webui.db.bak-20261009`): `model` row (`frank-zealot` → `frank-zealot`,
params + `meta.description` preserved), the `access_grant` row
(`model`/`frank-zealot`/`user`/`*`/`read`), and `openai.api_configs`
`model_ids` on all three backends; `systemctl --user restart
open-webui.service`. Family-account proof, in a real browser as `qa-e2e`:
the welcome screen selects `frank-zealot` and
`POST /api/v1/chat/completions {"model":"frank-zealot"}` answers
@@ -997,7 +955,6 @@
- **Live — Hermes re-keyed** (`config.yaml` `frank-dragoon` → `frank-dragoon`
in the provider block, `model.default` and the three `dankclaw-large`
auxiliary routes; backup `config.yaml.bak-20261009`; `hermes.service` +
`hermes-webui.service` restarted, gateway log clean, `/health` ok) and the
repo baseline `dankclaw/hermes/config.yaml.example` synced to the live
shape (context 262144). **Flagged, not fixed:** Hermes still routes its
`dankserver-small` auxiliary tasks at the id `frank-medium` on
@@ -1006,13 +963,10 @@
they need a routing decision (archon capacity), not a rename.
- **Deployer blind spot found by accident:** every
`deploy.sh dankserver --service frank-archon` since 2026-10-08 04:09
restarted the **staging worktree**, not the deploy checkout — a campaign
drop-in `frank-archon.service.d/bench.conf` overrode `Environment=` and
`ExecStart`. The health gate went green each time. Verified parity before
removing it (serve script + unit byte-identical, env identical except the
alias, staging HEAD an ancestor of `main`), moved the drop-in to
`/home/dank/.AGENTTRASH/20261009-frank-archon-bench-dropin/`, and
restarted: `/v1/models` then answered `frank-archon`. Post-deploy
behavioural proof is now part of the runbook.
- Repo: seams re-pinned (`frank_zealot`/`frank_dragoon`/`frank_archon`
unit bats), fleet harnesses retargeted (`bench-probe`, `deep-probe`,
@@ -1132,13 +1086,11 @@
(refusals gone), vision back (the v2 tower loads beside the checkpoint),
MTP head inside the checkpoint, 2×200,192 slots, `HALOGEN_SPEC_ADAPT=0`.
Runbook: `serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-swift15.md`.
**Incident (03:09:39–03:11:42):** the deploy started the unit while host
`MemAvailable` was ~74.7 GiB, 3.34 GiB short of the engine's pin-gate
requirement (62.11 GiB checkpoint + 16 GiB floor = 78.11 GiB); it
crash-looped 8 boots, each dying in ~5 s at the pin gate, and
self-resolved in ~2 min when the dying 27B engine freed RAM (boot #9
pinned clean at 88.4 GiB available). **Latent risk + fix:** the host was
left with only ~1.8 GiB `MemAvailable`, so a restart would crash-loop
again; `serve-frank-dragoon.sh` now has a host-RAM preflight that checks
`MemAvailable` against the pin gate BEFORE `podman run` and WAITS up to
`FRANK_DRAGOON_RAM_WAIT` (600 s) for the RAM to clear, refusing loudly if
@@ -1160,7 +1112,6 @@
`reasoning_content`; models.yml flipped accordingly). Container renamed
to the unit name (Frank's call; `frank-archon-qwen` → legacy list);
deployer image gate → `podman image exists` (local builds carry no
RepoDigests) and health gate reads the BODY — FreeToken 200s with
`{"status":"loading"}` mid-load, which false-greened a deploy at 6 s
(vLLM 503'd; the bare curl was honest there). Verified: FLEET-OK
completion, reasoning split (17×24 → "408"), native `tool_calls`, vision
@@ -1281,11 +1232,9 @@
7,653 slots, budget 31.29.
- Shape G v1 (0a4ff30) tried util 1.0: **both workers refused init before a
single weight loaded** — `ValueError: Free memory on device cuda:N
(31.29/31.86 GiB ...)`, NRestarts=2 crash loop, zero payload progress.
The CLI field is `Field(gt=0, le=1)`, but the worker demands free >
budget, so budget == total is a trap, not a knife-edge. Frank: ".982 is
the law, never 1.00" — now with a second proof.
- Shape G v2 (c67a2ae, deployed green incl. e2e 33/33): util back to
0.982, graph base to 0 (the shape F boot reported a NEGATIVE -0.06 GiB
capture delta, so the 256 MiB base was fiction; 0 + 12 MiB/graph still
prices a future 18-graph spec-decode shape at 0.21 GiB), KV_BYTES
@@ -1305,7 +1254,6 @@
## 2026-10-02
**frank-dragoon cutover: vLLM MiMo -> ROCmFPX Q8_0_ROCMFPX_AGENT + MTP**
- 2026-10-02 — frank-dragoon: vLLM MiMo → ROCmFPX `Q8_0_ROCMFPX_AGENT`
payload + MTP from the repo's own bundled head, one session at the native
262,144. See [[frank-dragoon]].
@@ -1324,7 +1272,6 @@
**frank-zealot: Q5_K_M, then three contexts + vision, sized by two boots**
- 2026-10-02 — frank-zealot: Q5_K_M payload, then three contexts + vision
in one day, sized by two boots (`CTX_SIZE=540672 PARALLEL=3`, f16 K +
q8_0 V under flash attention, `--mmproj`). See [[frank-zealot]].
@@ -1353,7 +1300,6 @@
**frank-archon shape D: 3 x 200704, fp8, pool sized for 2 contexts**
Frank's order: shrink frank-archon to a 200704-token window with about two
resident full-length contexts, cut the KV pool, and drop MNBT to 1024. The
first attempt (`30904ba`, 2 x 200704 + SEQS=2) **crash-looped**; the revert
@@ -1362,7 +1308,6 @@ top of the proven-good base, root cause first.
- **ROOT CAUSE: `--max-num-seqs 2` cannot compile this model. It was never
OOM, never MNBT, never the KV dtype.** Four single-variable boot probes
against the then-deployed shape (production stopped for each, candidate
booted AS the service): 262144 x SEQS 2 fails at BOTH MNBT rungs with
`torch.fx.experimental.symbolic_shapes.ConstraintViolationError:
Constraints violated (L['query_start_loc'].size()[0]) ... specialized it
@@ -1398,13 +1343,11 @@ top of the proven-good base, root cause first.
- **Harness contract followed the window:** `models.yml` `contextWindow`
262144 -> **200704**; side effect: frank-archon's compaction request fits
inside a frank-zealot slot again. A comment-only follow-up `295d578` was
deployed so the per-service revision marker sits at HEAD. Rollback for
the pre-shape-D fp8+MTP4 era stays git tag
`frank-archon-fp8-mtp-20261001`. Runbook [[frank-archon]].
**plex: the readiness gate probed HTTP against a TLS-only service**
Found by accident, and it had been latent since the secure-connections posture
was set: `deploy_plex` gated on `curl -fsS http://127.0.0.1:32400/identity`,
which **cannot** succeed. PMS drops plain HTTP off-loopback - the probe's
@@ -1428,18 +1371,14 @@ and then answers with nothing, and curl reports "Empty reply from server" (52).
got the same treatment. Fixed to `curl -fsSk --max-time 3
https://127.0.0.1:32400/identity` (`-k`: PMS serves its own self-signed
cert on 32400). Re-deployed `39f0e0d`: plex recorded deployed, e2e
33/33, frank-archon untouched (`health=200`, NRestarts=0).
- **Operating lesson: `deploy.sh <target>` with no `--service` is a
fleet-wide drain.** It restarts every service whose recorded revision is
behind HEAD and whose paths changed since. To touch one service, name it.
**frank-archon shape E: `--expert-precise-memory` (+1,026 arena slots), a mounted compile cache, and the P2P answer**
Shape D left one question open — the plan's overhead term was reserving 2.35 GiB
of compile headroom the boot never used — and the answer to Frank's free-VRAM
question arrived together with a boot-cost bug that had been costing 181 s on
every restart since the cutover.
- **`--expert-precise-memory` is real and it fits (option A).** Boot-probed on
the shape-D base with production stopped, the candidate booted AS the service
@@ -1463,11 +1402,9 @@ every restart since the cutover.
1.38x` — measured, rejected as the shipped value, kept as the conservative
fallback. With 3.15e9/rank the endpoint keeps shape D's 1.99x concurrency and
gains the arena, which is the shape D follow-up had actually been looking for.
- **THE COMPILE CACHE WAS BEING THROWN AWAY ON EVERY RESTART — 181 s each.**
Measured: a fresh container has **no `/root/.cache` at all** (`No such file or
directory`); after our boot it holds `/root/.cache/vllm` at
**1,154,714,333 B in 4,665 files** plus `/root/.cache/comgr` at 229 MB, inside
the ephemeral layer, and `podman rm -f` (every deploy, restart, reboot)
discards them. The engine names the loss itself: `Compiling a graph for compile
range (1, 1024) takes 173.19 s` ... `torch.compile took 181.20 s in total` ...
`init engine (profile, create kv cache, warmup model) took 217.26 s
@@ -1520,7 +1457,6 @@ every restart since the cutover.
not a crash — the container and its serve script must be polled by recorded id
and their logs dumped on the exit path, or a failure looks like a bad flag.
The flag was fine both times.
- Live: `health=200`, `NRestarts=0`, arena 17.22 GiB / 7,522 slots, KV 398,658
tokens / 1.99x. Docs: [[frank-archon]].
- **Deployed `1ac4d48`, e2e 33/33, and the live plan matched the probe exactly:**
`'expert_precise_memory': True`, `runtime overhead … 5.25 GiB`, `expert cache
@@ -1529,7 +1465,6 @@ every restart since the cutover.
precise overhead and the arena growth are now production numbers, not a
probe's projection.
- **The compile-cache fix was then proven on the same endpoint, two consecutive
boots of the deployed shape:** cold `Compiling a graph for compile range
(1, 1024) takes 179.24 s` → `torch.compile took 188.26 s`; warm `Directly load
the compiled graph(s) for compile range (1, 1024) from the cache, took
2.143 s` → `torch.compile took 3.97 s in total` → `init engine (profile,
@@ -1560,14 +1495,11 @@ every restart since the cutover.
**frank-archon RESTORED to fp8 3 x 262144 (bf16 boot fails)**
bf16 (--kv-cache-dtype auto) at 2 x 262144 did not stay up, with OR without
--expert-precise-memory:
- with precise: NRestarts 1->3, each attempt ~6.7 min then
`Deactivated successfully` (API server drains after an EngineCore fatal
error) — cause not captured, the serve script rm'd the container.
- without precise (default 7.60 GiB overhead, arena 14.3 GiB): NRestarts
climbing, container `Exited (1)`, and the journal finally showed the real
error:
`RuntimeError: Worker failed with error 'Constraints violated
@@ -1588,12 +1520,9 @@ Next: bisect bf16 vs SEQS in a window (bf16 + SEQS=3 isolates the dtype).
**frank-archon: --expert-precise-memory + bf16 = boot crash loop**
**Reverted.** The two changes Frank asked for were applied together and the
result crash-looped: `NRestarts` climbed 1 -> 2 -> 3, each attempt running
~6.7 minutes then logging `Deactivated successfully` (vLLM's API server drains
and exits 0 after an EngineCore fatal error, so `podman wait` returns 0; the
new `Restart=always` turned a silent death into a visible loop).
The budget arithmetic explains it without needing the error text, because the
arena is DEVICE-resident (the fp8 3-window shape's plan predicts 28.27 GiB and
@@ -1621,7 +1550,6 @@ by paying for it elsewhere (smaller pool, fewer sessions, or a shorter context).
**frank-archon: 2 bf16 windows + --expert-precise-memory**
Frank's final shape for the night. Two changes at once, both measured in
principle before the boot:
@@ -1647,7 +1575,6 @@ planned arena; it comes out of free VRAM instead.
**frank-dragoon: Q4_K_XL payload + 4 x 163840 sessions**
Frank's call. Payload swapped from ajgazin's **abliterated** Swift-1.5
UD-Q8_K_XL (29.3 GiB) to **unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL** (16.35 GiB,
sha256 3f227079…b01e, vanilla/Apache-2.0 — the endpoint is no longer
@@ -1669,7 +1596,6 @@ weights 16.35 + KV 40.0 + drafter 1.07 + linear state 0.56 + runtime ~4 =
**e2e gate flakiness (12-crawl) — do not panic at the banner**
`specs/12-crawl.spec.ts:23` ("every user open link reaches its app") failed the
deploy gate twice and then passed cleanly on a manual re-run:
- run 1 (deploy): `Expected "plex.franksdank.cloud"` / `Received "app.plex.tv"`
@@ -1688,7 +1614,6 @@ any repo service.
**frank-archon bf16 no-spec cutover**
- `/scripts/frank-archon.env` — bf16 no-spec cutover, FINAL shape after the
night ladder: MAXLEN **241664** x SEQS 3, KV_BYTES **10,184,000,000**/rank,
KV_DTYPE auto (bf16), MNBT 1024, GATHER CHUNKS=32/LANES=64, UTIL 0.98;
@@ -1725,7 +1650,6 @@ any repo service.
- **Client contract fixed (thinking + tools).** The live endpoint returned a
null `reasoning` field and dumped the thinking text INLINE in `content`
with a dangling `</think>` (measured: `'We need answer…Brief.\n</think>\n\n391'`)
— the malformed thinking block clients choke on — because the deployed argv
had no `--reasoning-parser`. The GLM rollback (`d91379d`) had dropped all
three contract flags and the qwen rollback inherited the gap. Restored from
`ecaad36`'s values, confirmed against this fork's registries:
@@ -1755,11 +1679,9 @@ any repo service.
knife-edge shape change, send a large-`max_tokens` request before calling it
done. FOUR slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93
GiB per rank for activations/graph capture. Frank's call 2026-10-01.
- `/scripts/frank-archon.service` — **Restart=always, not on-failure** (plus
`StartLimitIntervalSec=0` and `OOMScoreAdjust=-500`). A runtime worker fault
makes vLLM's API server exit CLEAN, so ExecStart returned 0, systemd logged
"Deactivated successfully", and on-failure NEVER fired: the endpoint stayed
down until a human noticed. An explicit `systemctl stop` still wins, so
bench windows and deploys are unaffected.
- **Final shape (Frank's call, 2026-10-01): 3 x 262144, pool at the FLOOR,
MNBT 2048, rest to the arena.** MAXLEN 262144 x SEQS 3 = 786,432 tokens of
@@ -1779,7 +1701,6 @@ any repo service.
**frank-dragoon cutover: gufo -> vLLM TheRock, MiMo-V2.6 BF16 + cross-model MTP**
- **Question that started it:** can a Qwen3.5 MTP drafter attach to
frank-zealot, and how does vLLM run the FULL BF16 MiMo checkpoint? Answer 1:
not on frank-zealot — SGLang reads the MTP head from the TARGET checkpoint
@@ -1793,7 +1714,6 @@ any repo service.
prefill / 512 decode):** base 17,876 t/s prefill / 13.1 t/s decode (13.1
is ~94% of the LPDDR5x BF16 roof - decode here is bandwidth, not a bug);
+DFlash-8 7,976 / 17.4 (acceptance 59%/26% per position); +MTP-2 7,380 /
19.8 (71%/44%, token rate 57.4%). Deployed MTP-2: +51% decode. Both spec
arms pay ~55-59% of prefill. `num_speculative_tokens` beyond 2 unmeasured.
- **Measured numerics law:** spec arms do NOT reproduce the no-spec greedy
text (bf16 batched verification reorders reductions; near-tie argmaxes
@@ -1804,9 +1724,7 @@ any repo service.
`reasoningContentField: "reasoning"`, and the extraBody moved to the
Qwen3.5-family recipe (0.7/0.8) with `chat_template_kwargs`
(a top-level `preserve_thinking` is ignored by vLLM).
- Deployer gap found pre-deploy: the spec-draft gate demanded a single
`model.safetensors` (the dflash layout) and would have rejected the
deployed MTP arm (4 shards behind an index); fixed to accept both
layouts, gated by bats. `z-lab/Qwen3.5-9B-DFlash`'s "vLLM PR #40898
unmerged" card claim is FALSE - `vllm/v1/spec_decode/dflash.py` ships in
0.30 (verified in-image); the drafter stays on disk as the alternative.
@@ -1845,7 +1763,6 @@ any repo service.
cutover), TP=2, 131,072 ctx x 2 seqs, bf16 KV, util 0.98, NBATCH 2048,
`--enable-expert-offload --expert-precise-memory`, prefix caching, vision +
`glm47` tool and reasoning parsers, MTP OFF, prefetch OFF.
- Measured on the deployed engine: **decode 9.9-10.0 tok/s, prefill 918 tok/s**
(warm, 2.6k prompt), KV 138,561 tokens; arena 14.51 GiB = 2,383 slots per
rank over 12,096 host-backed experts (73.64 GiB pinned per rank).
- **The wall — and a corrected one.** The first account of the shortfall
@@ -1951,7 +1868,6 @@ any repo service.
per-sequence KDA state and the MTP draft's KV, and `gpu_worker` caps KV at
that reserve, making a legal `--max-model-len` unbootable. Shipped
`scripts/patches/gpu_worker-kv-cap.patch` (lets an explicit operator KV
request raise the cap) — NOT needed by the deployed shape.
- Rollback: commit `2339fae` (qwen FP8 + MTP4); the qwen payload and the R9V
state dirs are intact on disk.
@@ -2131,7 +2047,6 @@ any repo service.
(`20260921-frank-dragoon-assembled` 88 G, `-redownload` 30 G) and never
purged. Total now staged on this host: **627 G**.
- Verified after the moves: frank-dragoon and frank-zealot both `active`,
both `/health` 200, and all three live assets (Swift target, DFlash2
drafter, Gemma4 target) still present.
- **frank-dragoon payload swapped to Swift-1.5 (uncensored)** (production
change, Dankclaw). Frank asked whether the UD Q8 of
@@ -2248,7 +2163,6 @@ any repo service.
8 s, drafter 2 s, session 3 s) against a 20-minute `LLM_WAIT` budget;
RSS 30.6 GiB; a live short-prompt request returned
`Server-Timing: ttft;dur=549.8, inter_token;dur=24.2` (~41 tok/s).
Verified live: `/health` 200, `/ready` 200, `/v1/models` →
`frank-dragoon` at `context_length 262144`, a chat completion carrying
`reasoning_content`, a `tool_calls` response with
`finish_reason: tool_calls`, and SSE streaming.
@@ -2315,9 +2229,6 @@ any repo service.
- **LANDMINE, not changed here:** `/home/dank/models` was cleared by Frank
mid-session to free space. frank-archon is still answering `:7770` from
mmapped-but-unlinked inodes (`/models` is empty inside the running
container) — **the endpoint cannot be restarted until the 175 GiB payload
is re-fetched.** A reboot or a container restart takes frank-archon down.
- Deploy-state reconciliation (Frank's call: no restart) — the autoresearch trio
(`f187165` PLE 1800, `41e9189` prefix caching, `ba2100e` docs) was found already
MERGED to main + pushed AND already live on frank-archon: the running
`frank-archon-r9v` container (booted 12:13 from the `ba2100e` checkout) carries
@@ -2330,9 +2241,7 @@ any repo service.
failure `11-admin` is the FrankBook-Pro Mac mini (`192.168.0.112:8989`) offline
(100 % ICMP loss + TCP timeout), the documented asleep-host condition, not a
regression.
- `scripts/frank-archon.env`, runbook — **prefix caching DEPLOYED on
frank-archon** (Frank's go-ahead; commit `41e9189`): E8-qualified, zero
VRAM cost, 9x/63x reuse TTFT. Verified on :7770 (argv flag, health 200,
completion "PREFIX-CACHE-LIVE"). Gotcha re-confirmed: the container is
in dank's ROOTLESS podman store (`sudo podman` inspects an empty store
and reports nothing); probe via the shim's `docker` or the unit MainPID.
@@ -2464,7 +2373,6 @@ any repo service.
## 2026-08-26
- `/dankclaw/Caddyfile` — Both Plex vhosts proxy to PMS over a TLS upstream (`insecure_skip_verify`); PMS drops plain-HTTP connections from non-loopback sources (empty-body 502s).
- `/dankclaw/gitea/quadlet/gitea-caddy.container` — Standalone Gitea Caddy quadlet retired; web proxy ownership moved to canonical Caddy; deploy-plan/deploy.sh/deploy-remote.sh + tests updated for the migration.
- Incident — Post-reboot: Caddy (a bare compose container — `restart: unless-stopped` does not survive shutdown's explicit stop) and Pi-hole (quadlet crash-loop, restart counter 1645) were both down. Root cause: `net.ipv4.ip_unprivileged_port_start` reset to 80 by a later-sorted sysctl drop-in AND a stale line in `/etc/sysctl.conf`, which systemd-sysctl applies AFTER all of `sysctl.d/`.
- `/AGENTS.md` — New MANDATORY "Boot Persistence & Self-Healing" section.
- `/dankclaw/caddy/quadlet/caddy.container`, `/dankclaw/caddy/env.example` — Caddy is now a boot-persistent quadlet (joins `dankclaw_default`, `Pull=never`, env from `/home/dank/Containers/caddy/env`); removed from compose.yaml.
- `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — Repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` stages conflicting drop-ins, comments out the stale sysctl.conf line, and applies `sysctl --system` (the boot code path).