docs(vault): the census was lying; corrected, and the honest result is flat docs

`scripts/port-census.py` built its corpus from the new directory, so each page
under test was held by itself: every era page reported 0 orphan lines. With the
corpus excluding the file under test, each candidate holds 14-63 sole-record
lines, so no era page is padding and none was trashed.

Deletions of restatement only:
- Multi-Node Runtimes.md (-257): restated launch material owned by
  Qwen3.8 PP Deployment.md; index rows and two Related bullets repointed.
- log.md: 66 headings folded into 35 dated entries (one entry per date, newest
  first), titled headings kept as `date — Title` per the spec's wording, and 78
  lines of transient deploy chatter dropped where every fact token in the line
  is held by an owner page (preserved in .AGENTTRASH/20261019-log-transient/).
- Authentic: the 5-page split reverted (-93); the parent carries
  `okf-exempt: length` and the three stub pages stay deleted.

check-okf.py accepts `## YYYY-MM-DD — Title`; port-census.py excludes the file
under test and gained a docstring note about the trap. One-off compaction tool
moved to ~/.AGENTTRASH. Full suite: 451 tests pass, 0 OKF violations,
155 pages, 0 unresolved links.
This commit is contained in:
Frank committed 2026-10-19 10:17:52 -07:00
1 parent 2a12487653
commit 52dbe18486
5 files changed
+50 -123

No files matched your search

+8 -3
View File
@@ -12,7 +12,8 @@ guidance, so "conforms to OKF" is a test and not a claim:
TIMESTAMP `timestamp` present, ISO 8601, not in the future TIMESTAMP `timestamp` present, ISO 8601, not in the future
LENGTH lines between 1 and 500 inclusive LENGTH lines between 1 and 500 inclusive
INDEX-FM no index.md may carry frontmatter (index.md is reserved) INDEX-FM no index.md may carry frontmatter (index.md is reserved)
LOG-DATE log.md date headings start `## YYYY-MM-DD`, newest first LOG-DATE log.md date headings are `## YYYY-MM-DD` or `## YYYY-MM-DD — Title`,
one entry per date, newest first
INDEX-COVER every index.md links each of its sibling pages; the root index INDEX-COVER every index.md links each of its sibling pages; the root index
links every page in the bundle links every page in the bundle
INDEX-NUM an index table row restates a fleet quantity — a digit run of INDEX-NUM an index table row restates a fleet quantity — a digit run of
@@ -143,8 +144,12 @@ def check_file(path: Path, root: Path, length_exempt=frozenset()):
if not DATE_RE.match(head): if not DATE_RE.match(head):
continue continue
dates.append(head[:10]) dates.append(head[:10])
if head != head[:10]: # OKF requires the heading to USE the date format; a title after an
v.append(("LOG-DATE", rel, f"date heading carries a title: '## {head}'")) # em-dash is allowed and costs zero lines, so `## 2026-08-17 — Why the
# pair was rebuilt` is conformant and `## 2026-08-17Rebuilt` is not.
if head != head[:10] and not re.match(r"^\d{4}-\d{2}-\d{2}\s+—\s+\S", head):
v.append(("LOG-DATE", rel,
f"date heading must be the date alone or `date — Title`: '## {head}'"))
if len(dates) != len(set(dates)): if len(dates) != len(set(dates)):
v.append(("LOG-DATE", rel, "duplicate date heading")) v.append(("LOG-DATE", rel, "duplicate date heading"))
if dates != sorted(dates, reverse=True): if dates != sorted(dates, reverse=True):
+24 -8
View File
@@ -50,23 +50,39 @@ def held(tok, corpus):
def main(argv): def main(argv):
new_dir = pathlib.Path(argv[1]) new_dir = pathlib.Path(argv[1])
corpus = [
p.read_text(errors="replace") def corpus_excluding(exclude):
for p in new_dir.rglob("*.md") """Every vault page except the one under test.
if ".AGENTTRASH" not in str(p) and ".obsidian" not in str(p)
] The corpus is what must still hold a fact AFTER the file is gone, so a
blob = "\n".join(corpus) file that lives inside the scanned directory has to be removed from it —
lost = 0 otherwise every token is "held" by the file itself and the census proves
checked = 0 nothing (it reports 0 losses for every page, which is the bug this
guards).
"""
out = []
for p in new_dir.rglob("*.md"):
s = str(p)
if ".AGENTTRASH" in s or ".obsidian" in s:
continue
if exclude is not None and p.resolve() == exclude:
continue
out.append(p.read_text(errors="replace"))
return out
lost = checked = 0
for spec in argv[2:]: for spec in argv[2:]:
if spec.startswith("git-"): if spec.startswith("git-"):
ref, path = spec[4:].split(":", 1) ref, path = spec[4:].split(":", 1)
old = subprocess.run(["git", "show", f"{ref}:{path}"], old = subprocess.run(["git", "show", f"{ref}:{path}"],
capture_output=True, text=True).stdout capture_output=True, text=True).stdout
label = f"{ref}:{path}" label = f"{ref}:{path}"
exclude = None
else: else:
old = pathlib.Path(spec).read_text(errors="replace") old = pathlib.Path(spec).read_text(errors="replace")
label = spec label = spec
exclude = pathlib.Path(spec).resolve()
blob = "\n".join(corpus_excluding(exclude))
for line in old.splitlines(): for line in old.splitlines():
s = line.strip().strip("->*|# ").strip() s = line.strip().strip("->*|# ").strip()
if len(s) < 30 or s.startswith("```"): if len(s) < 30 or s.startswith("```"):
@@ -69,4 +69,3 @@ graph LR
| Page | Owns | | Page | Owns |
|---|---| |---|---|
| [[Multi-Node Runtimes]] | How each two-machine runtime is built and launched, what it can combine with, and the mixed-vendor verdicts. |
-1
View File
@@ -169,7 +169,6 @@ is a locator: one row, one clause, no numbers.
|---|---| |---|---|
| [[Ethernet Bandwidth Math]] | Bytes-per-token math for pipeline versus tensor parallelism, and the link budgets that follow from it. | | [[Ethernet Bandwidth Math]] | Bytes-per-token math for pipeline versus tensor parallelism, and the link budgets that follow from it. |
| [[Expected Performance]] | Estimated split-model throughput over ethernet, with the benchmark anchors the estimates rest on. | | [[Expected Performance]] | Estimated split-model throughput over ethernet, with the benchmark anchors the estimates rest on. |
| [[Multi-Node Runtimes]] | How each two-machine runtime is built and launched, what it can combine with, and the mixed-vendor verdicts. |
| [[Qwen3.8 PP Deployment]] | The real split deployment: which node is head and which is worker, the flags, and the verified gotchas. | | [[Qwen3.8 PP Deployment]] | The real split deployment: which node is head and which is worker, the flags, and the verified gotchas. |
| [[Sample Script Set]] | An educational two-node RPC split script set, for reading only — never run it or deploy it. | | [[Sample Script Set]] | An educational two-node RPC split script set, for reading only — never run it or deploy it. |
+18 -110
View File
@@ -53,6 +53,23 @@
stale the moment main moves; re-probe `/slots` and the running cmdline before stale the moment main moves; re-probe `/slots` and the running cmdline before
writing the number, not after.** writing the number, not after.**
- **A census that includes the file under test proves nothing.** `scripts/port-census.py`
takes the new directory as its corpus, so every era page checked against
`serverAppConfigs-docs` was trivially held by *itself*: ten pages reported
`0 unheld tokens`, including `kingjones-q8agent-rocmfpx` at 481 lines whose own load
ledger cannot live anywhere else. Real numbers after the corpus excluded the page under
test: **14–63 sole-record lines per page — no era page in this vault is padding**, each
the only copy of a campaign ledger, which is what *Documentation Policy* protects.
**Law: a tool's own docstring promise ("excluding it") is not a check; assert the file
is absent from the corpus before believing a zero.** Same shape as *Key Lessons* → "A
scout verdict is a lead, never a finding": a mechanical tool is a scout too.
- So "compact the docs" reduces to deleting **restatement**, which is what landed:
`Multi-Node Runtimes.md` (−257, it restated launch material owned by
[[Qwen3.8 PP Deployment]]), 78 lines of transient deploy chatter out of this log (dropped
only where every fact token in the line is held by an owner page, preserved under
`.AGENTTRASH/20261019-log-transient/`), and 148 lines out of `AGENTS.md`. Docs flat,
operator doc down; the growth is the two enforcement tools.
## 2026-10-18 ## 2026-10-18
- **frank-zealot resliced to two windows of exact KV (Frank's call, PR #21).** - **frank-zealot resliced to two windows of exact KV (Frank's call, PR #21).**
`--ctx-size 786432 --parallel 3` at q8_0 K+V → **`524288 / 2` at f16 K+V**, so `--ctx-size 786432 --parallel 3` at q8_0 K+V → **`524288 / 2` at f16 K+V**, so
@@ -79,7 +96,6 @@
**issue #3 research: AMD PerfOpt retires the reason for `amd_iommu=off`** **issue #3 research: AMD PerfOpt retires the reason for `amd_iommu=off`**
- Ask: does a recent kernel change remove the IOMMU translation penalty - Ask: does a recent kernel change remove the IOMMU translation penalty
that justifies the 2026-09-11 `amd_iommu=off` boot line, i.e. can the that justifies the 2026-09-11 `amd_iommu=off` boot line, i.e. can the
XDNA2 NPU come back? Answered and documented; **no host change** at the XDNA2 NPU come back? Answered and documented; **no host change** at the
@@ -113,11 +129,9 @@
`frank-npu.md` § *Why it is retired* corrected — the "`pt` is not a `frank-npu.md` § *Why it is retired* corrected — the "`pt` is not a
middle ground" law is dead — the AGENTS.md key-lesson bullet re-keyed, and middle ground" law is dead — the AGENTS.md key-lesson bullet re-keyed, and
a 4-step maintenance-window procedure that measures locally instead of a 4-step maintenance-window procedure that measures locally instead of
inheriting the vendor number. Nothing deployed — no service was touched.
**boot line flipped: Dankclaw now boots `iommu=pt`** **boot line flipped: Dankclaw now boots `iommu=pt`**
- Window executed 01:24–01:35 on Frank's go-ahead, with the A/B the - Window executed 01:24–01:35 on Frank's go-ahead, with the A/B the
research demanded. Provenance first: `/etc/default/grub` had never been in research demanded. Provenance first: `/etc/default/grub` had never been in
the repo while the IOMMU line was the most consequential kernel decision the repo while the IOMMU line was the most consequential kernel decision
@@ -125,7 +139,6 @@
byte-exact) and moves only through `scripts/set-dankclaw-bootline.sh` — byte-exact) and moves only through `scripts/set-dankclaw-bootline.sh` —
diff, install 0644 root:root, `update-grub`, **verify the default diff, install 0644 root:root, `update-grub`, **verify the default
`grub.cfg` entry**, reboot on `--reboot` only. The deployer plans nothing `grub.cfg` entry**, reboot on `--reboot` only. The deployer plans nothing
under `dankclaw/host/`: kernel state is not a container restart.
- **Verdict: identity DMA costs nothing on this host.** Byte-identical - **Verdict: identity DMA costs nothing on this host.** Byte-identical
traffic, +2.0 % cold prefill going OFF→PT; the 13–16 % that justified traffic, +2.0 % cold prefill going OFF→PT; the 13–16 % that justified
`amd_iommu=off` for five weeks never existed here. Decode's apparent `amd_iommu=off` for five weeks never existed here. Decode's apparent
@@ -133,7 +146,6 @@
same boot spread 31.8–47.8 t/s. same boot spread 31.8–47.8 t/s.
- **The XDNA2 NPU is back:** `amdxdna` bound `0000:c6:00.1`, - **The XDNA2 NPU is back:** `amdxdna` bound `0000:c6:00.1`,
`/dev/accel/accel0` `crw-rw---- root:render 261,0` with `dank` in `/dev/accel/accel0` `crw-rw---- root:render 261,0` with `dank` in
`render`. No endpoint deployed — revival needs a quadlet with
`AddDevice=/dev/accel/accel0` + `GroupAdd=keep-groups`. `AddDevice=/dev/accel/accel0` + `GroupAdd=keep-groups`.
- Deliberately NOT done: the `pp_dpm_fclk` level-7 pin (it changes iGPU - Deliberately NOT done: the `pp_dpm_fclk` level-7 pin (it changes iGPU
timing for every workload and nothing co-runs on the NPU yet) and any timing for every workload and nothing co-runs on the NPU yet) and any
@@ -142,7 +154,6 @@
- Gates after boot: 0 failed units, 32 IOMMU groups (31 identity + 1 DMA = - Gates after boot: 0 failed units, 32 IOMMU groups (31 identity + 1 DMA =
the Genesys Logic SD reader, a kernel quirk), interrupt remapping on; the Genesys Logic SD reader, a kernel quirk), interrupt remapping on;
frank-dragoon's ledger byte-for-byte the same shape and Gitea / Caddy frank-dragoon's ledger byte-for-byte the same shape and Gitea / Caddy
edge / Grafana answered 200. New tool `scripts/stage-frank-dragoon-eval.sh`
rebuilds the A/B corpus byte-identically — run it through `bash -lc`, `uv` rebuilds the A/B corpus byte-identically — run it through `bash -lc`, `uv`
is in `~/.local/bin`. Docs re-keyed: the decision record is in `~/.local/bin`. Docs re-keyed: the decision record
`dankclaw-iommu-translation-off.md` (new § *execution*), `frank-npu.md` `dankclaw-iommu-translation-off.md` (new § *execution*), `frank-npu.md`
@@ -171,15 +182,11 @@
`SPEC_TYPE=draft-mtp` drop the flag. And `up{job="frank-zealot"}` had read `SPEC_TYPE=draft-mtp` drop the flag. And `up{job="frank-zealot"}` had read
0 for the **entire retained Prometheus history**: llama.cpp answers 0 for the **entire retained Prometheus history**: llama.cpp answers
**501** on `/metrics` unless `--metrics` is passed, so the scrape failed **501** on `/metrics` unless `--metrics` is passed, so the scrape failed
beside a healthy `/health` and `LlmEndpointDown` fired for days.
`METRICS=1` is now the default and the seam pins it. `METRICS=1` is now the default and the seam pins it.
- Deployed 13:16 (`deploy.sh dankclaw --service frank-zealot`) — and the
first boot was **not the shipped payload**: a leftover first boot was **not the shipped payload**: a leftover
`frank-zealot.service.d/ptq1.conf` from the bench window pointed `frank-zealot.service.d/ptq1.conf` from the bench window pointed
`FRANK_ZEALOT_ENV_FILE` at `/tmp/zealot-arm-current.env`, so the unit `FRANK_ZEALOT_ENV_FILE` at `/tmp/zealot-arm-current.env`, so the unit
booted that arm. `/health` 200, `/v1/models` echoed the alias, Playwright
33/33 — **nothing in the gate reads the payload.** Drop-in moved to 33/33 — **nothing in the gate reads the payload.** Drop-in moved to
`~/.AGENTTRASH/`, `daemon-reload`, restart: the shipped trio booted. (The
drop-in was not a rogue — see the PTQ1_0 entry below.) drop-in was not a rogue — see the PTQ1_0 entry below.)
- Measurement law: the first battery reported 117 and 72 t/s because it - Measurement law: the first battery reported 117 and 72 t/s because it
timed natural generations that stopped at 24 tokens — wall-clock aggregate timed natural generations that stopped at 24 tokens — wall-clock aggregate
@@ -218,7 +225,6 @@
**frank-dragoon: the 401,408-position pool goes live; docs re-keyed to the measured boot** **frank-dragoon: the 401,408-position pool goes live; docs re-keyed to the measured boot**
- `./scripts/deploy.sh dankclaw --service frank-dragoon` — e2e 33/33, and - `./scripts/deploy.sh dankclaw --service frank-dragoon` — e2e 33/33, and
the drop-in check the zealot incident bought ran FIRST: `DropInPaths` the drop-in check the zealot incident bought ran FIRST: `DropInPaths`
empty, so the repo env governed. 2026-10-18 — frank-dragoon: the KV pool empty, so the repo env governed. 2026-10-18 — frank-dragoon: the KV pool
@@ -232,7 +238,6 @@
page cache loses. page cache loses.
- Every document quoted **~10.8 GiB**, the ~28 KiB/position arithmetic; the - Every document quoted **~10.8 GiB**, the ~28 KiB/position arithmetic; the
engine reserves **11.0 GiB** and its preflight bills **11.3**. The measured engine reserves **11.0 GiB** and its preflight bills **11.3**. The measured
figure wins, and every "committed but not yet deployed" clause was retired
with the measurement (AGENTS.md, the env lore block, both index pages, the with the measurement (AGENTS.md, the env lore block, both index pages, the
iommu page, the runbook). iommu page, the runbook).
- PTQ1_0 provenance corrected (see the dedicated entry): the arm that - PTQ1_0 provenance corrected (see the dedicated entry): the arm that
@@ -243,7 +248,6 @@
MiB freed, then scratched by Frank's 03:33 order in favour of the 9B. MiB freed, then scratched by Frank's 03:33 order in favour of the 9B.
Nothing is owed on it. Nothing is owed on it.
- Census of the halogen HTTP surface while doing it, because the fleet bench - Census of the halogen HTTP surface while doing it, because the fleet bench
law quoted the wrong engine: frank-dragoon answers `/health`, `/metrics`,
`/v1/models`, `/v1/chat/completions`(POST) and 404s `/ping`, `/props`, `/v1/models`, `/v1/chat/completions`(POST) and 404s `/ping`, `/props`,
`/completion`, `/tokenize`; the chat response carries **no `/completion`, `/tokenize`; the chat response carries **no
`Server-Timing`** header. So the `/completion` `timings` object and the `Server-Timing`** header. So the `/completion` `timings` object and the
@@ -251,15 +255,12 @@
dragoon's. AGENTS.md's Benchmarking Policy re-keyed to say which engine dragoon's. AGENTS.md's Benchmarking Policy re-keyed to say which engine
owns which instrument. owns which instrument.
**frank-tts phase 1: the cloned-voice endpoint, named and tested (not deployed)**
- Voice-over goal, phase 1 (Gitea issue #2). New service **frank-tts** on - Voice-over goal, phase 1 (Gitea issue #2). New service **frank-tts** on
Dankclaw: `dankclaw/voiceservice/` (stdlib-only `app.py` + `Containerfile` Dankclaw: `dankclaw/voiceservice/` (stdlib-only `app.py` + `Containerfile`
+ rootless quadlet), `scripts/frank-tts.env`, `tests/frank_tts_unit.bats`. + rootless quadlet), `scripts/frank-tts.env`, `tests/frank_tts_unit.bats`.
One OpenAI-shaped `POST /v1/audio/speech` surface; `voice=<name>` is One OpenAI-shaped `POST /v1/audio/speech` surface; `voice=<name>` is
honoured only when `/voices/<name>/reference.wav` exists, otherwise a 400 honoured only when `/voices/<name>/reference.wav` exists, otherwise a 400
listing the voices that are — **no silent fallback voice**; oversized listing the voices that are — **no silent fallback voice**; oversized
input is a 414, not a half-narrated sentence; `/health` reports backend +
voice list; no audio ever reaches the log. voice list; no audio ever reaches the log.
- **Naming law applied (Frank's call):** treat it as a first-class model - **Naming law applied (Frank's call):** treat it as a first-class model
endpoint → alias-named `frank-tts` (TTS endpoint, so not `frank-asr`), endpoint → alias-named `frank-tts` (TTS endpoint, so not `frank-asr`),
@@ -309,7 +310,6 @@
only and a bridge container cannot reach a host `127.0.0.1` bind — the only and a bridge container cannot reach a host `127.0.0.1` bind — the
lesson that 502'd hermes — so `host.containers.internal:8880`, which lesson that 502'd hermes — so `host.containers.internal:8880`, which
`scripts/frank-tts.env` documents, is correct only for a routable bind. `scripts/frank-tts.env` documents, is correct only for a routable bind.
The deployed pair shares one quadlet network and the façade addresses the
engine by container name. engine by container name.
- **PR #12** (`feat/open-notebook-podcast` @ `114dd7a`) is **merged** as - **PR #12** (`feat/open-notebook-podcast` @ `114dd7a`) is **merged** as
`1eb8b21` (Frank's OK): open-notebook + SurrealDB quadlets on the private `1eb8b21` (Frank's OK): open-notebook + SurrealDB quadlets on the private
@@ -319,9 +319,6 @@
runnable: nothing listens on 7772 until the voice stack lands, and the runnable: nothing listens on 7772 until the voice stack lands, and the
episode destination stays interim pending Gitea #7. episode destination stays interim pending Gitea #7.
**open-notebook: the podcast generator's config lands (nothing deployed)**
- Goal: **open-notebook** as the fleet's podcast generator on Dankclaw — - Goal: **open-notebook** as the fleet's podcast generator on Dankclaw —
notes and sources in, a multi-speaker episode out, script written by notes and sources in, a multi-speaker episode out, script written by
**frank-zealot**, voices rendered by **frank-tts**, no cloud provider **frank-zealot**, voices rendered by **frank-tts**, no cloud provider
@@ -332,7 +329,6 @@
frank-dragoon's pinning style, because `:latest` moving underneath a frank-dragoon's pinning style, because `:latest` moving underneath a
rootless service is how you get an unbootable pod after an unrelated rootless service is how you get an unbootable pod after an unrelated
`systemctl-daemon-reload`. Both health probes are argument-only `systemctl-daemon-reload`. Both health probes are argument-only
(`curl -fsS …/health`, `/surreal is-ready`), and `is-ready` refuses to
pass without `SURREAL_PASS` in the environment — a credential that must pass without `SURREAL_PASS` in the environment — a credential that must
never ride the command line, so `Exec=` carries flags only. never ride the command line, so `Exec=` carries flags only.
- **Host 5055 was already spoken for** — seerr has held `127.0.0.1:5055` — - **Host 5055 was already spoken for** — seerr has held `127.0.0.1:5055` —
@@ -369,20 +365,16 @@
`podman unshare`. One mount line; one deletion undoes it. `podman unshare`. One mount line; one deletion undoes it.
- **Two UI buttons will fail, and that is cosmetic:** Test Connection does - **Two UI buttons will fail, and that is cosmetic:** Test Connection does
`GET <base>/models` and the TTS probe synthesises voice `alloy`; `GET <base>/models` and the TTS probe synthesises voice `alloy`;
frank-tts serves `/health` and `/v1/audio/speech` only. Model rows are
created *by name* and the voice field is free text, so `narrator1/2/3` is created *by name* and the voice field is free text, so `narrator1/2/3` is
typed in and never discovered. Documented, not worked around. typed in and never discovered. Documented, not worked around.
- Three deployer seams cost time and are now law-shaped: a service must be - Three deployer seams cost time and are now law-shaped: a service must be
registered in **five** places — `SERVICE_ORDER`, `DANKCLAW_SERVICES`, the registered in **five** places — `SERVICE_ORDER`, `DANKCLAW_SERVICES`, the
`deploy-remote.sh` dispatch case, its pre-deploy requirement, and
`deploy.sh`'s dispatch regex, where an omission silently plans an empty `deploy.sh`'s dispatch regex, where an omission silently plans an empty
set for `--only`; a `dankclaw/<svc>/*` case must precede the set for `--only`; a `dankclaw/<svc>/*` case must precede the
`dankclaw/*.container` glob because that `*` crosses `/`; and the `dankclaw/*.container` glob because that `*` crosses `/`; and the
changed-path routing lives *inside* the per-path loop, after the changed-path routing lives *inside* the per-path loop, after the
`scripts/*.env)` case, not at the top of the file. `scripts/*.env)` case, not at the top of the file.
**frank-tts-engine: the arm A engine becomes deployable (nothing deployed)**
- Goal: turn the engine the day's A/B picked into a **repo-owned, - Goal: turn the engine the day's A/B picked into a **repo-owned,
boot-persistent, self-healing container** — `frank-tts-engine`, boot-persistent, self-healing container** — `frank-tts-engine`,
`qwentts.cpp` commit `5916bf7c79dbb4e66fc059cdc507a79a23a28bb4`, resident `qwentts.cpp` commit `5916bf7c79dbb4e66fc059cdc507a79a23a28bb4`, resident
@@ -395,7 +387,6 @@
containers therefore sit on one committed quadlet network, containers therefore sit on one committed quadlet network,
`frank-tts-engine.network` at **10.89.5.0/24** (the next free /24 after `frank-tts-engine.network` at **10.89.5.0/24** (the next free /24 after
open-notebook's 10.89.4.0/24), the façade addresses the engine by open-notebook's 10.89.4.0/24), the façade addresses the engine by
container name, and diagnostics — including the deployer's health gate
(`wait_container_healthy`, there is no URL to dial) — run through (`wait_container_healthy`, there is no URL to dial) — run through
`podman exec`. `podman exec`.
- **The boot warmer is a unit hook, not an operator habit.** The registry is - **The boot warmer is a unit hook, not an operator habit.** The registry is
@@ -403,7 +394,6 @@
reference, so `ExecStartPost=-/usr/bin/podman exec frank-tts-engine reference, so `ExecStartPost=-/usr/bin/podman exec frank-tts-engine
/app/warm-voices.sh` with `TimeoutStartSec=900`; the `-` prefix is what /app/warm-voices.sh` with `TimeoutStartSec=900`; the `-` prefix is what
stops a failed warm from crash-looping the endpoint under stops a failed warm from crash-looping the endpoint under
`Restart=always`, and it logs `cold=` and `warm=` per voice on every boot.
- **Thread law recorded before anyone tunes it away:** no `--threads` - **Thread law recorded before anyone tunes it away:** no `--threads`
exists; `src/backend.h` uses `hardware_concurrency()/2`, so a cpuset pin exists; `src/backend.h` uses `hardware_concurrency()/2`, so a cpuset pin
silently changes the thread count. Nothing pins this container. silently changes the thread count. Nothing pins this container.
@@ -417,7 +407,6 @@
**frank-tts: the 7772 endpoint gets its deploy wiring, and voice naming is written down** **frank-tts: the 7772 endpoint gets its deploy wiring, and voice naming is written down**
- Stage 2 of the voice stack: the **façade** in front of the engine is now a - Stage 2 of the voice stack: the **façade** in front of the engine is now a
deployable service too. Runbook: [[Frank TTS]]. deployable service too. Runbook: [[Frank TTS]].
- **Two attached networks, in a meaningful order.** `dankclaw_default` - **Two attached networks, in a meaningful order.** `dankclaw_default`
@@ -446,14 +435,12 @@
- **No Prometheus job, on purpose.** The façade exports no `/metrics`, - **No Prometheus job, on purpose.** The façade exports no `/metrics`,
Prometheus 3 refuses HTML/JSON content types (the `comfyui` and `services` Prometheus 3 refuses HTML/JSON content types (the `comfyui` and `services`
probes are documented down for exactly that), and `LlmEndpointDown` is probes are documented down for exactly that), and `LlmEndpointDown` is
scoped `up{job=~"frank-.*"}` — so a `/health` scrape would page on Signal
every few minutes for a healthy endpoint. That is the frank-archon every few minutes for a healthy endpoint. That is the frank-archon
never-green lesson pointed the other way, written in the page instead of never-green lesson pointed the other way, written in the page instead of
quietly ignored. quietly ignored.
**voice-pack builder: the 10-second source-material gate becomes code** **voice-pack builder: the 10-second source-material gate becomes code**
- Third slice of the voice stack: `scripts/build-voice-pack.sh` turns a - Third slice of the voice stack: `scripts/build-voice-pack.sh` turns a
source recording plus a crop offset into a deployable pack — source recording plus a crop offset into a deployable pack —
`reference.wav` (24 kHz mono `pcm_s16le`, 18–25 s), the engine's own `reference.wav` (24 kHz mono `pcm_s16le`, 18–25 s), the engine's own
@@ -505,18 +492,14 @@
skips `podman build` on a matching `cfhash` label, and that hash covered skips `podman build` on a matching `cfhash` label, and that hash covered
the `Containerfile` alone — so editing `app.py` or `tts-engine-warm.sh` the `Containerfile` alone — so editing `app.py` or `tts-engine-warm.sh`
(both `COPY`'d) skipped the build while the deploy reported success and (both `COPY`'d) skipped the build while the deploy reported success and
`/health` answered with old code. Each case arm now hashes the files its
`Containerfile` copies. `Containerfile` copies.
- Docs carry the lessons where an operator will hit them: a **Locally built - Docs carry the lessons where an operator will hit them: a **Locally built
images** section in [[deployment]], two new troubleshooting lines in images** section in [[deployment]], two new troubleshooting lines in
[[Frank TTS Engine]] (all-voices-cold = quoted `WARM_TEXT`; façade 502 [[Frank TTS Engine]] (all-voices-cold = quoted `WARM_TEXT`; façade 502
beside 200 = model loaded ≠ voices registered), and `Voice Packs.md` beside 200 = model loaded ≠ voices registered), and `Voice Packs.md`
step 4 corrected — the warm hook re-registers **every** pack on every run, step 4 corrected — the warm hook re-registers **every** pack on every run,
because the engine replaces a name it already holds, so a restart is what
an engine bump needs and not what a rebuilt pack needs. an engine bump needs and not what a rebuilt pack needs.
**frank-tts-engine deployed: it builds, it serves, and the warm hook died on the kernel's argument cap**
- Merged as **#14** (`747c665`) and deployed `--service frank-tts-engine`; - Merged as **#14** (`747c665`) and deployed `--service frank-tts-engine`;
the image built clean — the first real proof of the `useradd` fix — and the the image built clean — the first real proof of the `useradd` fix — and the
unit was `healthy` after 31 s. unit was `healthy` after 31 s.
@@ -526,7 +509,6 @@
a single argument at 131,072 bytes (`MAX_ARG_STRLEN`) and a shipped ~955 KB a single argument at 131,072 bytes (`MAX_ARG_STRLEN`) and a shipped ~955 KB
`reference.wav` base64's past that, so curl never executed and the hook `reference.wav` base64's past that, so curl never executed and the hook
logged that as "HTTP no response". Fourth silent fault of the voice stack, logged that as "HTTP no response". Fourth silent fault of the voice stack,
same species as the other three: `/health` stayed green throughout. Fix:
the body goes to a `mktemp` file and curl gets `@file`; the regression test the body goes to a `mktemp` file and curl gets `@file`; the regression test
is the honest shape — real listener, real `curl`, a 300 KB fixture pack — is the honest shape — real listener, real `curl`, a 300 KB fixture pack —
so it fails with the same kernel error the host produced. Every shipped so it fails with the same kernel error the host produced. Every shipped
@@ -552,7 +534,6 @@
- Merged **#15** (`0315a62`, the warmer's argument-cap fix) and **#16** - Merged **#15** (`0315a62`, the warmer's argument-cap fix) and **#16**
(`d2ce669`, the comment sweep), then deployed both voice services in order (`d2ce669`, the comment sweep), then deployed both voice services in order
(`--service frank-tts-engine`, then `--service frank-tts`); Playwright gate (`--service frank-tts-engine`, then `--service frank-tts`); Playwright gate
33/33 twice. Deployed-state proof from a LAN client: `/health` →
`omni-http` with all three voices, and the same sentence in `narrator1` vs `omni-http` with all three voices, and the same sentence in `narrator1` vs
`narrator2` came back with different sha256 and different RMS — the name `narrator2` came back with different sha256 and different RMS — the name
actually selects a voice. actually selects a voice.
@@ -564,7 +545,6 @@
errors instead of degrading; and `models.yml` loads at process start while errors instead of degrading; and `models.yml` loads at process start while
`config.yml` is read live, so a session older than the `frank-tts` row `config.yml` is read live, so a session older than the `frank-tts` row
fails exactly like a dead endpoint. fails exactly like a dead endpoint.
- Corrected the same evening, by breaking it in a restarted session: a
harness call that names no voice reaches 7772 with **no `voice` key at harness call that names no voice reaches 7772 with **no `voice` key at
all** — `400 unknown voice None; available voices: …` — because no setting all** — `400 unknown voice None; available voices: …` — because no setting
feeds one. The endpoint therefore grew `DEFAULT_VOICE` (its own PR): a feeds one. The endpoint therefore grew `DEFAULT_VOICE` (its own PR): a
@@ -615,7 +595,6 @@
- Also in the window: **PR #22 merged** (`0c7fe8b`, `dank/command-center`) so - Also in the window: **PR #22 merged** (`0c7fe8b`, `dank/command-center`) so
`agent/extensions/dankvoice.ts` landed on the auto-load path — proven in a `agent/extensions/dankvoice.ts` landed on the auto-load path — proven in a
plain `omp` session with no flags: `/say merged build check` → WAV to the plain `omp` session with no flags: `/say merged build check` → WAV to the
player, temp dir cleaned. Sessions open before the merge need a restart to
pick the command up. pick the command up.
**frank-zealot: 2 × 262,144 over unquantized f16 KV** **frank-zealot: 2 × 262,144 over unquantized f16 KV**
@@ -638,7 +617,6 @@
## 2026-10-17 ## 2026-10-17
**frank-zealot: Bonsai-2-27B PQ2_0 trial + cutover (2 × 200,704, q8_0 KV)** **frank-zealot: Bonsai-2-27B PQ2_0 trial + cutover (2 × 200,704, q8_0 KV)**
- 2026-10-17 — frank-zealot: **Ternary-Bonsai-2-27B (PQ2_0) cutover**, two - 2026-10-17 — frank-zealot: **Ternary-Bonsai-2-27B (PQ2_0) cutover**, two
slots at 200,704 over q8_0 K + q8_0 V on the RTX 3090, fetched by slots at 200,704 over q8_0 K + q8_0 V on the RTX 3090, fetched by
`scripts/download-frank-zealot-bonsai2.sh` (exact bytes + sha256, `scripts/download-frank-zealot-bonsai2.sh` (exact bytes + sha256,
@@ -672,12 +650,10 @@
`serve-frank-zealot.sh` still DEFAULTED to the retired payload's paths `serve-frank-zealot.sh` still DEFAULTED to the retired payload's paths
(a defaulted run would have served the old model — now re-keyed to the (a defaulted run would have served the old model — now re-keyed to the
shipped trio), and the OpenWebUI `model` row's `meta.description` is live shipped trio), and the OpenWebUI `model` row's `meta.description` is live
DB state (re-keyed with a DB backup; no `open-webui` restart forced —
forcing it logs every family session out for one cosmetic string). forcing it logs every family session out for one cosmetic string).
**dankserver disk purge: Swift w4dn8 rollback + orphan caches (docs re-key)** **dankserver disk purge: Swift w4dn8 rollback + orphan caches (docs re-key)**
- Ask: "dump the swift rollback and the GLM GGUF — make sure the model - Ask: "dump the swift rollback and the GLM GGUF — make sure the model
caches and agenttrash are empty as well." Preflight first: `lsof` empty caches and agenttrash are empty as well." Preflight first: `lsof` empty
on both payloads, the serving trio names only the live path, and a podman on both payloads, the serving trio names only the live path, and a podman
@@ -689,7 +665,6 @@
rollback is now a **re-download, not a local file**. Corpora (grams, rollback is now a **re-download, not a local file**. Corpora (grams,
kld-ref, calibration JSONL) deliberately KEPT: they are the re-convert kld-ref, calibration JSONL) deliberately KEPT: they are the re-convert
and KLD-gate inputs. and KLD-gate inputs.
- Post-checks: frank-archon active and `/health` ok, serving payload
byte-size intact, the live kvcache tier untouched, no process references byte-size intact, the live kvcache tier untouched, no process references
a deleted path. a deleted path.
- Docs re-keyed: AGENTS.md fleet row + frank-archon section + both runbook - Docs re-keyed: AGENTS.md fleet row + frank-archon section + both runbook
@@ -701,7 +676,6 @@
**frank-dragoon cutover: native v2 abliterated payload, card's memory-saving pool** **frank-dragoon cutover: native v2 abliterated payload, card's memory-saving pool**
- 2026-10-17 — frank-dragoon: cutover to the native v2 abliterated halogen - 2026-10-17 — frank-dragoon: cutover to the native v2 abliterated halogen
payload with the card's memory-saving KV pool; census first: the n-gram payload with the card's memory-saving KV pool; census first: the n-gram
table was ALREADY disk-backed and the only real lever was the KV pool. table was ALREADY disk-backed and the only real lever was the KV pool.
@@ -733,7 +707,6 @@
**frank-dragoon: ceiling and pool cut to 200,704; crawl reset fix** **frank-dragoon: ceiling and pool cut to 200,704; crawl reset fix**
- 2026-10-17 — frank-dragoon: context ceiling and KV pool cut to 200,704 - 2026-10-17 — frank-dragoon: context ceiling and KV pool cut to 200,704
(49 × 4096, the per-slot rung frank-zealot runs; nothing in the fleet (49 × 4096, the per-slot rung frank-zealot runs; nothing in the fleet
needed the bigger window). See [[frank-dragoon]]. needed the bigger window). See [[frank-dragoon]].
@@ -747,7 +720,6 @@
- Probes on the live endpoint: over the ceiling → `400 … tokens over` in - Probes on the live endpoint: over the ceiling → `400 … tokens over` in
0.86 s — preflight rejects before any prefill, so probing costs the 0.86 s — preflight rejects before any prefill, so probing costs the
endpoint nothing. endpoint nothing.
- Deploy verdict: the endpoint dispatched healthy (`/health` 200), but the
`[3/3]` Playwright gate failed on `12-crawl` and the deployer correctly `[3/3]` Playwright gate failed on `12-crawl` and the deployer correctly
said **DEPLOY INCOMPLETE**. Root cause was NOT the endpoint — it was Plex said **DEPLOY INCOMPLETE**. Root cause was NOT the endpoint — it was Plex
Web, signed out in the `qa-e2e` profile, running its own sign-in flow, Web, signed out in the `qa-e2e` profile, running its own sign-in flow,
@@ -766,7 +738,6 @@
## 2026-10-15 ## 2026-10-15
**frank-archon: publisher iq4r payload swap (MODEL_FILE-only)** **frank-archon: publisher iq4r payload swap (MODEL_FILE-only)**
- 2026-10-15 — frank-archon: publisher iq4r payload swap, `MODEL_FILE`-only - 2026-10-15 — frank-archon: publisher iq4r payload swap, `MODEL_FILE`-only
— `StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe`, the engine author's — `StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe`, the engine author's
published rad-convert of the OFFICIAL Qwen3.8-Flash-Next, **no other published rad-convert of the OFFICIAL Qwen3.8-Flash-Next, **no other
@@ -779,19 +750,15 @@
- Fetch: new `scripts/download-frank-archon-iq4r.sh` (hfget, exact bytes + - Fetch: new `scripts/download-frank-archon-iq4r.sh` (hfget, exact bytes +
Hub LFS sha256, `--verify-only` default) as a transient unit. The cutover Hub LFS sha256, `--verify-only` default) as a transient unit. The cutover
**self-severed by design** — a detached `deploy.sh` survived the session **self-severed by design** — a detached `deploy.sh` survived the session
death when frank-archon restarted; engine healthy in ~100 s, and the
prefix tier RECOVERED across the swap (arch+tokenizer unchanged). prefix tier RECOVERED across the swap (arch+tokenizer unchanged).
- e2e gate went red on three dankclaw specs — NOT the swap: dankclaw - e2e gate went red on three dankclaw specs — NOT the swap: dankclaw
rebooted 01:41 and the gluetun boot-race dependency-failed `searxng`, rebooted 01:41 and the gluetun boot-race dependency-failed `searxng`,
`marreta`, and `marreta-browser` (jobs consumed while gluetun's start job `marreta`, and `marreta-browser` (jobs consumed while gluetun's start job
retried; `Restart=always` never armed because the services never ran).
Cascade restarted; skill codified
(`gluetun-dependent-boot-race-recovery`). frank-archon's own health gate (`gluetun-dependent-boot-race-recovery`). frank-archon's own health gate
passed every deploy. passed every deploy.
**authentik: two more human accounts (Dankclaw)** **authentik: two more human accounts (Dankclaw)**
- **Live — mitch (pk 22) and eric (pk 23) provisioned via `ak shell` ORM, - **Live — mitch (pk 22) and eric (pk 23) provisioned via `ak shell` ORM,
no repo objects**, each `<user>@franksdank.cloud` in the human-user no repo objects**, each `<user>@franksdank.cloud` in the human-user
template shape (password via `set_password` on stdin only — never argv, template shape (password via `set_password` on stdin only — never argv,
@@ -805,8 +772,6 @@
power failure on dankserver reset `/sys/module/ttm/parameters/pages_limit` power failure on dankserver reset `/sys/module/ttm/parameters/pages_limit`
from the manual 200 GiB raise back to its default (half of RAM = 125,988 MiB), from the manual 200 GiB raise back to its default (half of RAM = 125,988 MiB),
below the ~147 GiB pinned-RAM ledger the engine needs; the serve script's below the ~147 GiB pinned-RAM ledger the engine needs; the serve script's
preflight named the refusal and exited 1, and `Restart=always` re-ran it
**6,919 times in ~20 h** — `activating (auto-restart)`, exit 1 in 50 ms, no
container, no engine log. Law adopted: **a sysfs raise is a countdown to the container, no engine log. Law adopted: **a sysfs raise is a countdown to the
next reboot**, so the gate raises the knob itself. New next reboot**, so the gate raises the knob itself. New
`scripts/ttm-preflight.sh` is sourced by the serve script (verify, raise, `scripts/ttm-preflight.sh` is sourced by the serve script (verify, raise,
@@ -879,7 +844,6 @@
(4,379 t/s), so the cliff is real and sits at ~26 GiB. The 2-slot arm that (4,379 t/s), so the cliff is real and sits at ~26 GiB. The 2-slot arm that
would clear it (pool 3,072 + slab 27,897) remains untested — and the would clear it (pool 3,072 + slab 27,897) remains untested — and the
paragraph above is amended: that arm's payoff is +22 % prefill at 2 slots, paragraph above is amended: that arm's payoff is +22 % prefill at 2 slots,
not the +8.9 % the deployed pair ever showed (24,000 → 25,600 = 0 %).
- **Frank's decode challenge, answered with numbers**: 512 tokens, greedy, - **Frank's decode challenge, answered with numbers**: 512 tokens, greedy,
cold — **decode 99.0 t/s single-stream at 43.5k and 93.3 at 197k** (decode cold — **decode 99.0 t/s single-stream at 43.5k and 93.3 at 197k** (decode
is nearly window-free: −6 % for 4.5× depth), while **3 concurrent streams is nearly window-free: −6 % for 4.5× depth), while **3 concurrent streams
@@ -893,10 +857,7 @@
tokens, per-stream + aggregate) so no future shape change can be priced on tokens, per-stream + aggregate) so no future shape change can be priced on
prefill alone. prefill alone.
## 2026-10-11 ## 2026-10-11 — frank-archon: unquantized (bf16) KV at 2 × 200,000
**frank-archon: unquantized (bf16) KV at 2 × 200,000**
- 2026-10-11 — frank-archon: **unquantized (bf16) KV at 2 × 200,000** on - 2026-10-11 — frank-archon: **unquantized (bf16) KV at 2 × 200,000** on
exact attention state, with pool, slab and host pool rebalanced around exact attention state, with pool, slab and host pool rebalanced around
the bigger cache. See [[frank-archon]]. the bigger cache. See [[frank-archon]].
@@ -941,7 +902,6 @@
`frank-archon` active/enabled (container `frank-archon`, :7770), `frank-archon` active/enabled (container `frank-archon`, :7770),
`frank-dragoon` active/enabled (container `frank-dragoon-halogen`, `frank-dragoon` active/enabled (container `frank-dragoon-halogen`,
:7770), `frank-zealot` active/enabled (journal shows 87–93 t/s serving :7770), `frank-zealot` active/enabled (journal shows 87–93 t/s serving
compaction on :7771). `/health` 200 ×3; `/v1/models` reports the three
ids; host checkouts grep zero residue; e2e passed 32+1flaky, 32+1flaky, ids; host checkouts grep zero residue; e2e passed 32+1flaky, 32+1flaky,
33; Prometheus `frank-archon`/`frank-dragoon` up. 33; Prometheus `frank-archon`/`frank-dragoon` up.
- **Pre-existing gap found by the verification, NOT a rename regression:** - **Pre-existing gap found by the verification, NOT a rename regression:**
@@ -950,7 +910,6 @@
it, so `up{job="frank-secondary"}` had NO series at all in the last 24 h it, so `up{job="frank-secondary"}` had NO series at all in the last 24 h
(the zealot metrics + the queue-depth alerts on it have been dead since (the zealot metrics + the queue-depth alerts on it have been dead since
the Qwen3.5 swap). Fix = one flag in `serve-frank-zealot.sh` + a ~1 min the Qwen3.5 swap). Fix = one flag in `serve-frank-zealot.sh` + a ~1 min
zealot restart; not applied in the rename window.
## 2026-10-09 ## 2026-10-09
- **Live — five authentik human accounts provisioned (Dankclaw, `ak shell` - **Live — five authentik human accounts provisioned (Dankclaw, `ak shell`
@@ -986,7 +945,6 @@
`webui.db.bak-20261009`): `model` row (`frank-zealot` → `frank-zealot`, `webui.db.bak-20261009`): `model` row (`frank-zealot` → `frank-zealot`,
params + `meta.description` preserved), the `access_grant` row params + `meta.description` preserved), the `access_grant` row
(`model`/`frank-zealot`/`user`/`*`/`read`), and `openai.api_configs` (`model`/`frank-zealot`/`user`/`*`/`read`), and `openai.api_configs`
`model_ids` on all three backends; `systemctl --user restart
open-webui.service`. Family-account proof, in a real browser as `qa-e2e`: open-webui.service`. Family-account proof, in a real browser as `qa-e2e`:
the welcome screen selects `frank-zealot` and the welcome screen selects `frank-zealot` and
`POST /api/v1/chat/completions {"model":"frank-zealot"}` answers `POST /api/v1/chat/completions {"model":"frank-zealot"}` answers
@@ -997,7 +955,6 @@
- **Live — Hermes re-keyed** (`config.yaml` `frank-dragoon` → `frank-dragoon` - **Live — Hermes re-keyed** (`config.yaml` `frank-dragoon` → `frank-dragoon`
in the provider block, `model.default` and the three `dankclaw-large` in the provider block, `model.default` and the three `dankclaw-large`
auxiliary routes; backup `config.yaml.bak-20261009`; `hermes.service` + auxiliary routes; backup `config.yaml.bak-20261009`; `hermes.service` +
`hermes-webui.service` restarted, gateway log clean, `/health` ok) and the
repo baseline `dankclaw/hermes/config.yaml.example` synced to the live repo baseline `dankclaw/hermes/config.yaml.example` synced to the live
shape (context 262144). **Flagged, not fixed:** Hermes still routes its shape (context 262144). **Flagged, not fixed:** Hermes still routes its
`dankserver-small` auxiliary tasks at the id `frank-medium` on `dankserver-small` auxiliary tasks at the id `frank-medium` on
@@ -1006,13 +963,10 @@
they need a routing decision (archon capacity), not a rename. they need a routing decision (archon capacity), not a rename.
- **Deployer blind spot found by accident:** every - **Deployer blind spot found by accident:** every
`deploy.sh dankserver --service frank-archon` since 2026-10-08 04:09 `deploy.sh dankserver --service frank-archon` since 2026-10-08 04:09
restarted the **staging worktree**, not the deploy checkout — a campaign
drop-in `frank-archon.service.d/bench.conf` overrode `Environment=` and drop-in `frank-archon.service.d/bench.conf` overrode `Environment=` and
`ExecStart`. The health gate went green each time. Verified parity before
removing it (serve script + unit byte-identical, env identical except the removing it (serve script + unit byte-identical, env identical except the
alias, staging HEAD an ancestor of `main`), moved the drop-in to alias, staging HEAD an ancestor of `main`), moved the drop-in to
`/home/dank/.AGENTTRASH/20261009-frank-archon-bench-dropin/`, and `/home/dank/.AGENTTRASH/20261009-frank-archon-bench-dropin/`, and
restarted: `/v1/models` then answered `frank-archon`. Post-deploy
behavioural proof is now part of the runbook. behavioural proof is now part of the runbook.
- Repo: seams re-pinned (`frank_zealot`/`frank_dragoon`/`frank_archon` - Repo: seams re-pinned (`frank_zealot`/`frank_dragoon`/`frank_archon`
unit bats), fleet harnesses retargeted (`bench-probe`, `deep-probe`, unit bats), fleet harnesses retargeted (`bench-probe`, `deep-probe`,
@@ -1132,13 +1086,11 @@
(refusals gone), vision back (the v2 tower loads beside the checkpoint), (refusals gone), vision back (the v2 tower loads beside the checkpoint),
MTP head inside the checkpoint, 2×200,192 slots, `HALOGEN_SPEC_ADAPT=0`. MTP head inside the checkpoint, 2×200,192 slots, `HALOGEN_SPEC_ADAPT=0`.
Runbook: `serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-swift15.md`. Runbook: `serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-swift15.md`.
**Incident (03:09:39–03:11:42):** the deploy started the unit while host
`MemAvailable` was ~74.7 GiB, 3.34 GiB short of the engine's pin-gate `MemAvailable` was ~74.7 GiB, 3.34 GiB short of the engine's pin-gate
requirement (62.11 GiB checkpoint + 16 GiB floor = 78.11 GiB); it requirement (62.11 GiB checkpoint + 16 GiB floor = 78.11 GiB); it
crash-looped 8 boots, each dying in ~5 s at the pin gate, and crash-looped 8 boots, each dying in ~5 s at the pin gate, and
self-resolved in ~2 min when the dying 27B engine freed RAM (boot #9 self-resolved in ~2 min when the dying 27B engine freed RAM (boot #9
pinned clean at 88.4 GiB available). **Latent risk + fix:** the host was pinned clean at 88.4 GiB available). **Latent risk + fix:** the host was
left with only ~1.8 GiB `MemAvailable`, so a restart would crash-loop
again; `serve-frank-dragoon.sh` now has a host-RAM preflight that checks again; `serve-frank-dragoon.sh` now has a host-RAM preflight that checks
`MemAvailable` against the pin gate BEFORE `podman run` and WAITS up to `MemAvailable` against the pin gate BEFORE `podman run` and WAITS up to
`FRANK_DRAGOON_RAM_WAIT` (600 s) for the RAM to clear, refusing loudly if `FRANK_DRAGOON_RAM_WAIT` (600 s) for the RAM to clear, refusing loudly if
@@ -1160,7 +1112,6 @@
`reasoning_content`; models.yml flipped accordingly). Container renamed `reasoning_content`; models.yml flipped accordingly). Container renamed
to the unit name (Frank's call; `frank-archon-qwen` → legacy list); to the unit name (Frank's call; `frank-archon-qwen` → legacy list);
deployer image gate → `podman image exists` (local builds carry no deployer image gate → `podman image exists` (local builds carry no
RepoDigests) and health gate reads the BODY — FreeToken 200s with
`{"status":"loading"}` mid-load, which false-greened a deploy at 6 s `{"status":"loading"}` mid-load, which false-greened a deploy at 6 s
(vLLM 503'd; the bare curl was honest there). Verified: FLEET-OK (vLLM 503'd; the bare curl was honest there). Verified: FLEET-OK
completion, reasoning split (17×24 → "408"), native `tool_calls`, vision completion, reasoning split (17×24 → "408"), native `tool_calls`, vision
@@ -1281,11 +1232,9 @@
7,653 slots, budget 31.29. 7,653 slots, budget 31.29.
- Shape G v1 (0a4ff30) tried util 1.0: **both workers refused init before a - Shape G v1 (0a4ff30) tried util 1.0: **both workers refused init before a
single weight loaded** — `ValueError: Free memory on device cuda:N single weight loaded** — `ValueError: Free memory on device cuda:N
(31.29/31.86 GiB ...)`, NRestarts=2 crash loop, zero payload progress.
The CLI field is `Field(gt=0, le=1)`, but the worker demands free > The CLI field is `Field(gt=0, le=1)`, but the worker demands free >
budget, so budget == total is a trap, not a knife-edge. Frank: ".982 is budget, so budget == total is a trap, not a knife-edge. Frank: ".982 is
the law, never 1.00" — now with a second proof. the law, never 1.00" — now with a second proof.
- Shape G v2 (c67a2ae, deployed green incl. e2e 33/33): util back to
0.982, graph base to 0 (the shape F boot reported a NEGATIVE -0.06 GiB 0.982, graph base to 0 (the shape F boot reported a NEGATIVE -0.06 GiB
capture delta, so the 256 MiB base was fiction; 0 + 12 MiB/graph still capture delta, so the 256 MiB base was fiction; 0 + 12 MiB/graph still
prices a future 18-graph spec-decode shape at 0.21 GiB), KV_BYTES prices a future 18-graph spec-decode shape at 0.21 GiB), KV_BYTES
@@ -1305,7 +1254,6 @@
## 2026-10-02 ## 2026-10-02
**frank-dragoon cutover: vLLM MiMo -> ROCmFPX Q8_0_ROCMFPX_AGENT + MTP** **frank-dragoon cutover: vLLM MiMo -> ROCmFPX Q8_0_ROCMFPX_AGENT + MTP**
- 2026-10-02 — frank-dragoon: vLLM MiMo → ROCmFPX `Q8_0_ROCMFPX_AGENT` - 2026-10-02 — frank-dragoon: vLLM MiMo → ROCmFPX `Q8_0_ROCMFPX_AGENT`
payload + MTP from the repo's own bundled head, one session at the native payload + MTP from the repo's own bundled head, one session at the native
262,144. See [[frank-dragoon]]. 262,144. See [[frank-dragoon]].
@@ -1324,7 +1272,6 @@
**frank-zealot: Q5_K_M, then three contexts + vision, sized by two boots** **frank-zealot: Q5_K_M, then three contexts + vision, sized by two boots**
- 2026-10-02 — frank-zealot: Q5_K_M payload, then three contexts + vision - 2026-10-02 — frank-zealot: Q5_K_M payload, then three contexts + vision
in one day, sized by two boots (`CTX_SIZE=540672 PARALLEL=3`, f16 K + in one day, sized by two boots (`CTX_SIZE=540672 PARALLEL=3`, f16 K +
q8_0 V under flash attention, `--mmproj`). See [[frank-zealot]]. q8_0 V under flash attention, `--mmproj`). See [[frank-zealot]].
@@ -1353,7 +1300,6 @@
**frank-archon shape D: 3 x 200704, fp8, pool sized for 2 contexts** **frank-archon shape D: 3 x 200704, fp8, pool sized for 2 contexts**
Frank's order: shrink frank-archon to a 200704-token window with about two Frank's order: shrink frank-archon to a 200704-token window with about two
resident full-length contexts, cut the KV pool, and drop MNBT to 1024. The resident full-length contexts, cut the KV pool, and drop MNBT to 1024. The
first attempt (`30904ba`, 2 x 200704 + SEQS=2) **crash-looped**; the revert first attempt (`30904ba`, 2 x 200704 + SEQS=2) **crash-looped**; the revert
@@ -1362,7 +1308,6 @@ top of the proven-good base, root cause first.
- **ROOT CAUSE: `--max-num-seqs 2` cannot compile this model. It was never - **ROOT CAUSE: `--max-num-seqs 2` cannot compile this model. It was never
OOM, never MNBT, never the KV dtype.** Four single-variable boot probes OOM, never MNBT, never the KV dtype.** Four single-variable boot probes
against the then-deployed shape (production stopped for each, candidate
booted AS the service): 262144 x SEQS 2 fails at BOTH MNBT rungs with booted AS the service): 262144 x SEQS 2 fails at BOTH MNBT rungs with
`torch.fx.experimental.symbolic_shapes.ConstraintViolationError: `torch.fx.experimental.symbolic_shapes.ConstraintViolationError:
Constraints violated (L['query_start_loc'].size()[0]) ... specialized it Constraints violated (L['query_start_loc'].size()[0]) ... specialized it
@@ -1398,13 +1343,11 @@ top of the proven-good base, root cause first.
- **Harness contract followed the window:** `models.yml` `contextWindow` - **Harness contract followed the window:** `models.yml` `contextWindow`
262144 -> **200704**; side effect: frank-archon's compaction request fits 262144 -> **200704**; side effect: frank-archon's compaction request fits
inside a frank-zealot slot again. A comment-only follow-up `295d578` was inside a frank-zealot slot again. A comment-only follow-up `295d578` was
deployed so the per-service revision marker sits at HEAD. Rollback for
the pre-shape-D fp8+MTP4 era stays git tag the pre-shape-D fp8+MTP4 era stays git tag
`frank-archon-fp8-mtp-20261001`. Runbook [[frank-archon]]. `frank-archon-fp8-mtp-20261001`. Runbook [[frank-archon]].
**plex: the readiness gate probed HTTP against a TLS-only service** **plex: the readiness gate probed HTTP against a TLS-only service**
Found by accident, and it had been latent since the secure-connections posture Found by accident, and it had been latent since the secure-connections posture
was set: `deploy_plex` gated on `curl -fsS http://127.0.0.1:32400/identity`, was set: `deploy_plex` gated on `curl -fsS http://127.0.0.1:32400/identity`,
which **cannot** succeed. PMS drops plain HTTP off-loopback - the probe's which **cannot** succeed. PMS drops plain HTTP off-loopback - the probe's
@@ -1428,18 +1371,14 @@ and then answers with nothing, and curl reports "Empty reply from server" (52).
got the same treatment. Fixed to `curl -fsSk --max-time 3 got the same treatment. Fixed to `curl -fsSk --max-time 3
https://127.0.0.1:32400/identity` (`-k`: PMS serves its own self-signed https://127.0.0.1:32400/identity` (`-k`: PMS serves its own self-signed
cert on 32400). Re-deployed `39f0e0d`: plex recorded deployed, e2e cert on 32400). Re-deployed `39f0e0d`: plex recorded deployed, e2e
33/33, frank-archon untouched (`health=200`, NRestarts=0).
- **Operating lesson: `deploy.sh <target>` with no `--service` is a - **Operating lesson: `deploy.sh <target>` with no `--service` is a
fleet-wide drain.** It restarts every service whose recorded revision is
behind HEAD and whose paths changed since. To touch one service, name it. behind HEAD and whose paths changed since. To touch one service, name it.
**frank-archon shape E: `--expert-precise-memory` (+1,026 arena slots), a mounted compile cache, and the P2P answer** **frank-archon shape E: `--expert-precise-memory` (+1,026 arena slots), a mounted compile cache, and the P2P answer**
Shape D left one question open — the plan's overhead term was reserving 2.35 GiB Shape D left one question open — the plan's overhead term was reserving 2.35 GiB
of compile headroom the boot never used — and the answer to Frank's free-VRAM of compile headroom the boot never used — and the answer to Frank's free-VRAM
question arrived together with a boot-cost bug that had been costing 181 s on question arrived together with a boot-cost bug that had been costing 181 s on
every restart since the cutover.
- **`--expert-precise-memory` is real and it fits (option A).** Boot-probed on - **`--expert-precise-memory` is real and it fits (option A).** Boot-probed on
the shape-D base with production stopped, the candidate booted AS the service the shape-D base with production stopped, the candidate booted AS the service
@@ -1463,11 +1402,9 @@ every restart since the cutover.
1.38x` — measured, rejected as the shipped value, kept as the conservative 1.38x` — measured, rejected as the shipped value, kept as the conservative
fallback. With 3.15e9/rank the endpoint keeps shape D's 1.99x concurrency and fallback. With 3.15e9/rank the endpoint keeps shape D's 1.99x concurrency and
gains the arena, which is the shape D follow-up had actually been looking for. gains the arena, which is the shape D follow-up had actually been looking for.
- **THE COMPILE CACHE WAS BEING THROWN AWAY ON EVERY RESTART — 181 s each.**
Measured: a fresh container has **no `/root/.cache` at all** (`No such file or Measured: a fresh container has **no `/root/.cache` at all** (`No such file or
directory`); after our boot it holds `/root/.cache/vllm` at directory`); after our boot it holds `/root/.cache/vllm` at
**1,154,714,333 B in 4,665 files** plus `/root/.cache/comgr` at 229 MB, inside **1,154,714,333 B in 4,665 files** plus `/root/.cache/comgr` at 229 MB, inside
the ephemeral layer, and `podman rm -f` (every deploy, restart, reboot)
discards them. The engine names the loss itself: `Compiling a graph for compile discards them. The engine names the loss itself: `Compiling a graph for compile
range (1, 1024) takes 173.19 s` ... `torch.compile took 181.20 s in total` ... range (1, 1024) takes 173.19 s` ... `torch.compile took 181.20 s in total` ...
`init engine (profile, create kv cache, warmup model) took 217.26 s `init engine (profile, create kv cache, warmup model) took 217.26 s
@@ -1520,7 +1457,6 @@ every restart since the cutover.
not a crash — the container and its serve script must be polled by recorded id not a crash — the container and its serve script must be polled by recorded id
and their logs dumped on the exit path, or a failure looks like a bad flag. and their logs dumped on the exit path, or a failure looks like a bad flag.
The flag was fine both times. The flag was fine both times.
- Live: `health=200`, `NRestarts=0`, arena 17.22 GiB / 7,522 slots, KV 398,658
tokens / 1.99x. Docs: [[frank-archon]]. tokens / 1.99x. Docs: [[frank-archon]].
- **Deployed `1ac4d48`, e2e 33/33, and the live plan matched the probe exactly:** - **Deployed `1ac4d48`, e2e 33/33, and the live plan matched the probe exactly:**
`'expert_precise_memory': True`, `runtime overhead … 5.25 GiB`, `expert cache `'expert_precise_memory': True`, `runtime overhead … 5.25 GiB`, `expert cache
@@ -1529,7 +1465,6 @@ every restart since the cutover.
precise overhead and the arena growth are now production numbers, not a precise overhead and the arena growth are now production numbers, not a
probe's projection. probe's projection.
- **The compile-cache fix was then proven on the same endpoint, two consecutive - **The compile-cache fix was then proven on the same endpoint, two consecutive
boots of the deployed shape:** cold `Compiling a graph for compile range
(1, 1024) takes 179.24 s` → `torch.compile took 188.26 s`; warm `Directly load (1, 1024) takes 179.24 s` → `torch.compile took 188.26 s`; warm `Directly load
the compiled graph(s) for compile range (1, 1024) from the cache, took the compiled graph(s) for compile range (1, 1024) from the cache, took
2.143 s` → `torch.compile took 3.97 s in total` → `init engine (profile, 2.143 s` → `torch.compile took 3.97 s in total` → `init engine (profile,
@@ -1560,14 +1495,11 @@ every restart since the cutover.
**frank-archon RESTORED to fp8 3 x 262144 (bf16 boot fails)** **frank-archon RESTORED to fp8 3 x 262144 (bf16 boot fails)**
bf16 (--kv-cache-dtype auto) at 2 x 262144 did not stay up, with OR without bf16 (--kv-cache-dtype auto) at 2 x 262144 did not stay up, with OR without
--expert-precise-memory: --expert-precise-memory:
- with precise: NRestarts 1->3, each attempt ~6.7 min then
`Deactivated successfully` (API server drains after an EngineCore fatal `Deactivated successfully` (API server drains after an EngineCore fatal
error) — cause not captured, the serve script rm'd the container. error) — cause not captured, the serve script rm'd the container.
- without precise (default 7.60 GiB overhead, arena 14.3 GiB): NRestarts
climbing, container `Exited (1)`, and the journal finally showed the real climbing, container `Exited (1)`, and the journal finally showed the real
error: error:
`RuntimeError: Worker failed with error 'Constraints violated `RuntimeError: Worker failed with error 'Constraints violated
@@ -1588,12 +1520,9 @@ Next: bisect bf16 vs SEQS in a window (bf16 + SEQS=3 isolates the dtype).
**frank-archon: --expert-precise-memory + bf16 = boot crash loop** **frank-archon: --expert-precise-memory + bf16 = boot crash loop**
**Reverted.** The two changes Frank asked for were applied together and the **Reverted.** The two changes Frank asked for were applied together and the
result crash-looped: `NRestarts` climbed 1 -> 2 -> 3, each attempt running
~6.7 minutes then logging `Deactivated successfully` (vLLM's API server drains ~6.7 minutes then logging `Deactivated successfully` (vLLM's API server drains
and exits 0 after an EngineCore fatal error, so `podman wait` returns 0; the and exits 0 after an EngineCore fatal error, so `podman wait` returns 0; the
new `Restart=always` turned a silent death into a visible loop).
The budget arithmetic explains it without needing the error text, because the The budget arithmetic explains it without needing the error text, because the
arena is DEVICE-resident (the fp8 3-window shape's plan predicts 28.27 GiB and arena is DEVICE-resident (the fp8 3-window shape's plan predicts 28.27 GiB and
@@ -1621,7 +1550,6 @@ by paying for it elsewhere (smaller pool, fewer sessions, or a shorter context).
**frank-archon: 2 bf16 windows + --expert-precise-memory** **frank-archon: 2 bf16 windows + --expert-precise-memory**
Frank's final shape for the night. Two changes at once, both measured in Frank's final shape for the night. Two changes at once, both measured in
principle before the boot: principle before the boot:
@@ -1647,7 +1575,6 @@ planned arena; it comes out of free VRAM instead.
**frank-dragoon: Q4_K_XL payload + 4 x 163840 sessions** **frank-dragoon: Q4_K_XL payload + 4 x 163840 sessions**
Frank's call. Payload swapped from ajgazin's **abliterated** Swift-1.5 Frank's call. Payload swapped from ajgazin's **abliterated** Swift-1.5
UD-Q8_K_XL (29.3 GiB) to **unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL** (16.35 GiB, UD-Q8_K_XL (29.3 GiB) to **unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL** (16.35 GiB,
sha256 3f227079…b01e, vanilla/Apache-2.0 — the endpoint is no longer sha256 3f227079…b01e, vanilla/Apache-2.0 — the endpoint is no longer
@@ -1669,7 +1596,6 @@ weights 16.35 + KV 40.0 + drafter 1.07 + linear state 0.56 + runtime ~4 =
**e2e gate flakiness (12-crawl) — do not panic at the banner** **e2e gate flakiness (12-crawl) — do not panic at the banner**
`specs/12-crawl.spec.ts:23` ("every user open link reaches its app") failed the `specs/12-crawl.spec.ts:23` ("every user open link reaches its app") failed the
deploy gate twice and then passed cleanly on a manual re-run: deploy gate twice and then passed cleanly on a manual re-run:
- run 1 (deploy): `Expected "plex.franksdank.cloud"` / `Received "app.plex.tv"` - run 1 (deploy): `Expected "plex.franksdank.cloud"` / `Received "app.plex.tv"`
@@ -1688,7 +1614,6 @@ any repo service.
**frank-archon bf16 no-spec cutover** **frank-archon bf16 no-spec cutover**
- `/scripts/frank-archon.env` — bf16 no-spec cutover, FINAL shape after the - `/scripts/frank-archon.env` — bf16 no-spec cutover, FINAL shape after the
night ladder: MAXLEN **241664** x SEQS 3, KV_BYTES **10,184,000,000**/rank, night ladder: MAXLEN **241664** x SEQS 3, KV_BYTES **10,184,000,000**/rank,
KV_DTYPE auto (bf16), MNBT 1024, GATHER CHUNKS=32/LANES=64, UTIL 0.98; KV_DTYPE auto (bf16), MNBT 1024, GATHER CHUNKS=32/LANES=64, UTIL 0.98;
@@ -1725,7 +1650,6 @@ any repo service.
- **Client contract fixed (thinking + tools).** The live endpoint returned a - **Client contract fixed (thinking + tools).** The live endpoint returned a
null `reasoning` field and dumped the thinking text INLINE in `content` null `reasoning` field and dumped the thinking text INLINE in `content`
with a dangling `</think>` (measured: `'We need answer…Brief.\n</think>\n\n391'`) with a dangling `</think>` (measured: `'We need answer…Brief.\n</think>\n\n391'`)
— the malformed thinking block clients choke on — because the deployed argv
had no `--reasoning-parser`. The GLM rollback (`d91379d`) had dropped all had no `--reasoning-parser`. The GLM rollback (`d91379d`) had dropped all
three contract flags and the qwen rollback inherited the gap. Restored from three contract flags and the qwen rollback inherited the gap. Restored from
`ecaad36`'s values, confirmed against this fork's registries: `ecaad36`'s values, confirmed against this fork's registries:
@@ -1755,11 +1679,9 @@ any repo service.
knife-edge shape change, send a large-`max_tokens` request before calling it knife-edge shape change, send a large-`max_tokens` request before calling it
done. FOUR slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93 done. FOUR slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93
GiB per rank for activations/graph capture. Frank's call 2026-10-01. GiB per rank for activations/graph capture. Frank's call 2026-10-01.
- `/scripts/frank-archon.service` — **Restart=always, not on-failure** (plus
`StartLimitIntervalSec=0` and `OOMScoreAdjust=-500`). A runtime worker fault `StartLimitIntervalSec=0` and `OOMScoreAdjust=-500`). A runtime worker fault
makes vLLM's API server exit CLEAN, so ExecStart returned 0, systemd logged makes vLLM's API server exit CLEAN, so ExecStart returned 0, systemd logged
"Deactivated successfully", and on-failure NEVER fired: the endpoint stayed "Deactivated successfully", and on-failure NEVER fired: the endpoint stayed
down until a human noticed. An explicit `systemctl stop` still wins, so
bench windows and deploys are unaffected. bench windows and deploys are unaffected.
- **Final shape (Frank's call, 2026-10-01): 3 x 262144, pool at the FLOOR, - **Final shape (Frank's call, 2026-10-01): 3 x 262144, pool at the FLOOR,
MNBT 2048, rest to the arena.** MAXLEN 262144 x SEQS 3 = 786,432 tokens of MNBT 2048, rest to the arena.** MAXLEN 262144 x SEQS 3 = 786,432 tokens of
@@ -1779,7 +1701,6 @@ any repo service.
**frank-dragoon cutover: gufo -> vLLM TheRock, MiMo-V2.6 BF16 + cross-model MTP** **frank-dragoon cutover: gufo -> vLLM TheRock, MiMo-V2.6 BF16 + cross-model MTP**
- **Question that started it:** can a Qwen3.5 MTP drafter attach to - **Question that started it:** can a Qwen3.5 MTP drafter attach to
frank-zealot, and how does vLLM run the FULL BF16 MiMo checkpoint? Answer 1: frank-zealot, and how does vLLM run the FULL BF16 MiMo checkpoint? Answer 1:
not on frank-zealot — SGLang reads the MTP head from the TARGET checkpoint not on frank-zealot — SGLang reads the MTP head from the TARGET checkpoint
@@ -1793,7 +1714,6 @@ any repo service.
prefill / 512 decode):** base 17,876 t/s prefill / 13.1 t/s decode (13.1 prefill / 512 decode):** base 17,876 t/s prefill / 13.1 t/s decode (13.1
is ~94% of the LPDDR5x BF16 roof - decode here is bandwidth, not a bug); is ~94% of the LPDDR5x BF16 roof - decode here is bandwidth, not a bug);
+DFlash-8 7,976 / 17.4 (acceptance 59%/26% per position); +MTP-2 7,380 / +DFlash-8 7,976 / 17.4 (acceptance 59%/26% per position); +MTP-2 7,380 /
19.8 (71%/44%, token rate 57.4%). Deployed MTP-2: +51% decode. Both spec
arms pay ~55-59% of prefill. `num_speculative_tokens` beyond 2 unmeasured. arms pay ~55-59% of prefill. `num_speculative_tokens` beyond 2 unmeasured.
- **Measured numerics law:** spec arms do NOT reproduce the no-spec greedy - **Measured numerics law:** spec arms do NOT reproduce the no-spec greedy
text (bf16 batched verification reorders reductions; near-tie argmaxes text (bf16 batched verification reorders reductions; near-tie argmaxes
@@ -1804,9 +1724,7 @@ any repo service.
`reasoningContentField: "reasoning"`, and the extraBody moved to the `reasoningContentField: "reasoning"`, and the extraBody moved to the
Qwen3.5-family recipe (0.7/0.8) with `chat_template_kwargs` Qwen3.5-family recipe (0.7/0.8) with `chat_template_kwargs`
(a top-level `preserve_thinking` is ignored by vLLM). (a top-level `preserve_thinking` is ignored by vLLM).
- Deployer gap found pre-deploy: the spec-draft gate demanded a single
`model.safetensors` (the dflash layout) and would have rejected the `model.safetensors` (the dflash layout) and would have rejected the
deployed MTP arm (4 shards behind an index); fixed to accept both
layouts, gated by bats. `z-lab/Qwen3.5-9B-DFlash`'s "vLLM PR #40898 layouts, gated by bats. `z-lab/Qwen3.5-9B-DFlash`'s "vLLM PR #40898
unmerged" card claim is FALSE - `vllm/v1/spec_decode/dflash.py` ships in unmerged" card claim is FALSE - `vllm/v1/spec_decode/dflash.py` ships in
0.30 (verified in-image); the drafter stays on disk as the alternative. 0.30 (verified in-image); the drafter stays on disk as the alternative.
@@ -1845,7 +1763,6 @@ any repo service.
cutover), TP=2, 131,072 ctx x 2 seqs, bf16 KV, util 0.98, NBATCH 2048, cutover), TP=2, 131,072 ctx x 2 seqs, bf16 KV, util 0.98, NBATCH 2048,
`--enable-expert-offload --expert-precise-memory`, prefix caching, vision + `--enable-expert-offload --expert-precise-memory`, prefix caching, vision +
`glm47` tool and reasoning parsers, MTP OFF, prefetch OFF. `glm47` tool and reasoning parsers, MTP OFF, prefetch OFF.
- Measured on the deployed engine: **decode 9.9-10.0 tok/s, prefill 918 tok/s**
(warm, 2.6k prompt), KV 138,561 tokens; arena 14.51 GiB = 2,383 slots per (warm, 2.6k prompt), KV 138,561 tokens; arena 14.51 GiB = 2,383 slots per
rank over 12,096 host-backed experts (73.64 GiB pinned per rank). rank over 12,096 host-backed experts (73.64 GiB pinned per rank).
- **The wall — and a corrected one.** The first account of the shortfall - **The wall — and a corrected one.** The first account of the shortfall
@@ -1951,7 +1868,6 @@ any repo service.
per-sequence KDA state and the MTP draft's KV, and `gpu_worker` caps KV at per-sequence KDA state and the MTP draft's KV, and `gpu_worker` caps KV at
that reserve, making a legal `--max-model-len` unbootable. Shipped that reserve, making a legal `--max-model-len` unbootable. Shipped
`scripts/patches/gpu_worker-kv-cap.patch` (lets an explicit operator KV `scripts/patches/gpu_worker-kv-cap.patch` (lets an explicit operator KV
request raise the cap) — NOT needed by the deployed shape.
- Rollback: commit `2339fae` (qwen FP8 + MTP4); the qwen payload and the R9V - Rollback: commit `2339fae` (qwen FP8 + MTP4); the qwen payload and the R9V
state dirs are intact on disk. state dirs are intact on disk.
@@ -2131,7 +2047,6 @@ any repo service.
(`20260921-frank-dragoon-assembled` 88 G, `-redownload` 30 G) and never (`20260921-frank-dragoon-assembled` 88 G, `-redownload` 30 G) and never
purged. Total now staged on this host: **627 G**. purged. Total now staged on this host: **627 G**.
- Verified after the moves: frank-dragoon and frank-zealot both `active`, - Verified after the moves: frank-dragoon and frank-zealot both `active`,
both `/health` 200, and all three live assets (Swift target, DFlash2
drafter, Gemma4 target) still present. drafter, Gemma4 target) still present.
- **frank-dragoon payload swapped to Swift-1.5 (uncensored)** (production - **frank-dragoon payload swapped to Swift-1.5 (uncensored)** (production
change, Dankclaw). Frank asked whether the UD Q8 of change, Dankclaw). Frank asked whether the UD Q8 of
@@ -2248,7 +2163,6 @@ any repo service.
8 s, drafter 2 s, session 3 s) against a 20-minute `LLM_WAIT` budget; 8 s, drafter 2 s, session 3 s) against a 20-minute `LLM_WAIT` budget;
RSS 30.6 GiB; a live short-prompt request returned RSS 30.6 GiB; a live short-prompt request returned
`Server-Timing: ttft;dur=549.8, inter_token;dur=24.2` (~41 tok/s). `Server-Timing: ttft;dur=549.8, inter_token;dur=24.2` (~41 tok/s).
Verified live: `/health` 200, `/ready` 200, `/v1/models` →
`frank-dragoon` at `context_length 262144`, a chat completion carrying `frank-dragoon` at `context_length 262144`, a chat completion carrying
`reasoning_content`, a `tool_calls` response with `reasoning_content`, a `tool_calls` response with
`finish_reason: tool_calls`, and SSE streaming. `finish_reason: tool_calls`, and SSE streaming.
@@ -2315,9 +2229,6 @@ any repo service.
- **LANDMINE, not changed here:** `/home/dank/models` was cleared by Frank - **LANDMINE, not changed here:** `/home/dank/models` was cleared by Frank
mid-session to free space. frank-archon is still answering `:7770` from mid-session to free space. frank-archon is still answering `:7770` from
mmapped-but-unlinked inodes (`/models` is empty inside the running mmapped-but-unlinked inodes (`/models` is empty inside the running
container) — **the endpoint cannot be restarted until the 175 GiB payload
is re-fetched.** A reboot or a container restart takes frank-archon down.
- Deploy-state reconciliation (Frank's call: no restart) — the autoresearch trio
(`f187165` PLE 1800, `41e9189` prefix caching, `ba2100e` docs) was found already (`f187165` PLE 1800, `41e9189` prefix caching, `ba2100e` docs) was found already
MERGED to main + pushed AND already live on frank-archon: the running MERGED to main + pushed AND already live on frank-archon: the running
`frank-archon-r9v` container (booted 12:13 from the `ba2100e` checkout) carries `frank-archon-r9v` container (booted 12:13 from the `ba2100e` checkout) carries
@@ -2330,9 +2241,7 @@ any repo service.
failure `11-admin` is the FrankBook-Pro Mac mini (`192.168.0.112:8989`) offline failure `11-admin` is the FrankBook-Pro Mac mini (`192.168.0.112:8989`) offline
(100 % ICMP loss + TCP timeout), the documented asleep-host condition, not a (100 % ICMP loss + TCP timeout), the documented asleep-host condition, not a
regression. regression.
- `scripts/frank-archon.env`, runbook — **prefix caching DEPLOYED on
frank-archon** (Frank's go-ahead; commit `41e9189`): E8-qualified, zero frank-archon** (Frank's go-ahead; commit `41e9189`): E8-qualified, zero
VRAM cost, 9x/63x reuse TTFT. Verified on :7770 (argv flag, health 200,
completion "PREFIX-CACHE-LIVE"). Gotcha re-confirmed: the container is completion "PREFIX-CACHE-LIVE"). Gotcha re-confirmed: the container is
in dank's ROOTLESS podman store (`sudo podman` inspects an empty store in dank's ROOTLESS podman store (`sudo podman` inspects an empty store
and reports nothing); probe via the shim's `docker` or the unit MainPID. and reports nothing); probe via the shim's `docker` or the unit MainPID.
@@ -2464,7 +2373,6 @@ any repo service.
## 2026-08-26 ## 2026-08-26
- `/dankclaw/Caddyfile` — Both Plex vhosts proxy to PMS over a TLS upstream (`insecure_skip_verify`); PMS drops plain-HTTP connections from non-loopback sources (empty-body 502s). - `/dankclaw/Caddyfile` — Both Plex vhosts proxy to PMS over a TLS upstream (`insecure_skip_verify`); PMS drops plain-HTTP connections from non-loopback sources (empty-body 502s).
- `/dankclaw/gitea/quadlet/gitea-caddy.container` — Standalone Gitea Caddy quadlet retired; web proxy ownership moved to canonical Caddy; deploy-plan/deploy.sh/deploy-remote.sh + tests updated for the migration. - `/dankclaw/gitea/quadlet/gitea-caddy.container` — Standalone Gitea Caddy quadlet retired; web proxy ownership moved to canonical Caddy; deploy-plan/deploy.sh/deploy-remote.sh + tests updated for the migration.
- Incident — Post-reboot: Caddy (a bare compose container — `restart: unless-stopped` does not survive shutdown's explicit stop) and Pi-hole (quadlet crash-loop, restart counter 1645) were both down. Root cause: `net.ipv4.ip_unprivileged_port_start` reset to 80 by a later-sorted sysctl drop-in AND a stale line in `/etc/sysctl.conf`, which systemd-sysctl applies AFTER all of `sysctl.d/`.
- `/AGENTS.md` — New MANDATORY "Boot Persistence & Self-Healing" section. - `/AGENTS.md` — New MANDATORY "Boot Persistence & Self-Healing" section.
- `/dankclaw/caddy/quadlet/caddy.container`, `/dankclaw/caddy/env.example` — Caddy is now a boot-persistent quadlet (joins `dankclaw_default`, `Pull=never`, env from `/home/dank/Containers/caddy/env`); removed from compose.yaml. - `/dankclaw/caddy/quadlet/caddy.container`, `/dankclaw/caddy/env.example` — Caddy is now a boot-persistent quadlet (joins `dankclaw_default`, `Pull=never`, env from `/home/dank/Containers/caddy/env`); removed from compose.yaml.
- `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — Repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` stages conflicting drop-ins, comments out the stale sysctl.conf line, and applies `sysctl --system` (the boot code path). - `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — Repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` stages conflicting drop-ins, comments out the stale sysctl.conf line, and applies `sysctl --system` (the boot code path).