dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

llm: retire the trio service names — fleet services renamed to frank-archon/frank-dragoon/frank-zealot

Browse Source 
The 2026-10-09 wire aliases become the service names everywhere:
systemd units, serve scripts, env files and FRANK_* variables,
deploy-plan/deploy-remote service keys, container names, Prometheus
job names + alert rules, Grafana dashboard queries, bats seams,
runbook filenames, homepage/AGENTS lore, and historical prose.
Rollback tags renamed in lockstep. Residue in the tree: zero.
Retired frank-medium/small/npu names are history and stay.

Known casualty: scripts/patches/*.patch context lines were rewritten;
those retired-era vLLM-fork patches no longer apply byte-exact.
This commit is contained in:
Frank
2026-10-10 17:03:14 -07:00
parent 5ced474eb9
commit 55f87fcdce
195 changed files with 2085 additions and 2064 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/log.md
+157 -137
View File
@@ -1,5 +1,25 @@
# Bundle Change Log
## 2026-10-10
- **Fleet service rename: the StarCraft names became the SERVICE names.**
The 2026-10-09 wire alias moved onto every repo artifact: units, serve
scripts, env files and their `FRANK_*` variables, deployer service keys,
container names, Prometheus/Grafana job names, bats seams, runbook
filenames and all historical prose. `frank-archon` (dankserver :7770),
`frank-dragoon` (dankclaw :7770), `frank-zealot` (dankclaw :7771). The
three former service names are fully retired: `git grep` over the tree
returns zero, rollback tags were renamed in lockstep (e.g. tag
`frank-archon-freetoken-pr587-20261005`), and the harness `models.yml`
lore matches. Host cutover = stop old unit + remove old container,
deploy the renamed service (the deployer relinks the unit under the new
name; `add_pending_services` auto-trashes the retired markers). The
retired `frank-medium`/`frank-small`/`frank-npu` names are historical
and stay as they are.
- **Known casualty:** the two vendored vLLM-fork patches under
`scripts/patches/` carry the old service names in context lines; the
sweep rewrote them, so those patches (retired eras, rebuildable from
the renamed tags) no longer apply byte-exact.
## 2026-10-09
- **Live — five authentik human accounts provisioned (Dankclaw, `ak shell`
ORM, no repo objects):** kirby (pk 17), ry (18), nic (19), debo (20),
@@ -20,18 +40,18 @@
that makes the picker model visible to every `role='user'` account.
- **Live — the Protoss rename finished on the wire (all three endpoints).**
Served model ids moved `frank-primary` → **frank-archon**
(`FRANK_PRIMARY_SERVED_NAME`), `frank-large` → **frank-dragoon**
(`HALOGEN_MODEL_ID`), `frank-secondary` → **frank-zealot** (`ALIAS`). The
Served model ids moved `frank-archon` → **frank-archon**
(`FRANK_ARCHON_SERVED_NAME`), `frank-dragoon` → **frank-dragoon**
(`HALOGEN_MODEL_ID`), `frank-zealot` → **frank-zealot** (`ALIAS`). The
services, units, scripts, env files and runbook pages keep the old names —
container/unit identity stays stable, only the alias moved. Deploys in order:
`dankclaw --service frank-secondary` (e2e gate 33/33 green), `dankclaw
--service frank-large` (healthy 30 s), `dankserver --service frank-primary`
`dankclaw --service frank-zealot` (e2e gate 33/33 green), `dankclaw
--service frank-dragoon` (healthy 30 s), `dankserver --service frank-archon`
(healthy 102 s), `dankclaw --service homepage`. Verified on `/v1/models`,
not on chat echoes: `frank-zealot` (`n_ctx 200704`, Q6_K), `frank-dragoon`
(`context_length 262144`), `frank-archon`.
- **Live — OpenWebUI re-keyed in the same window** (snapshot
`webui.db.bak-20261009`): `model` row (`frank-secondary` → `frank-zealot`,
`webui.db.bak-20261009`): `model` row (`frank-zealot` → `frank-zealot`,
params + `meta.description` preserved), the `access_grant` row
(`model`/`frank-zealot`/`user`/`*`/`read`), and `openai.api_configs`
`model_ids` on all three backends; `systemctl --user restart
@@ -42,27 +62,27 @@
probe returns `{"items":[],"total":0}` for that account while the picker
is plainly populated — the API census is not the acceptance test, the UI
snapshot is.
- **Live — Hermes re-keyed** (`config.yaml` `frank-large` → `frank-dragoon`
- **Live — Hermes re-keyed** (`config.yaml` `frank-dragoon` → `frank-dragoon`
in the provider block, `model.default` and the three `dankclaw-large`
auxiliary routes; backup `config.yaml.bak-20261009`; `hermes.service` +
`hermes-webui.service` restarted, gateway log clean, `/health` ok) and the
repo baseline `dankclaw/hermes/config.yaml.example` synced to the live
shape (context 262144). **Flagged, not fixed:** Hermes still routes its
`dankserver-small` auxiliary tasks at the id `frank-medium` on
`192.168.0.235:7770`, an endpoint that has served `frank-primary` since
`192.168.0.235:7770`, an endpoint that has served `frank-archon` since
2026-09-20 — those routes were dead before this rename and remain dead;
they need a routing decision (archon capacity), not a rename.
- **Deployer blind spot found by accident:** every
`deploy.sh dankserver --service frank-primary` since 2026-10-08 04:09
`deploy.sh dankserver --service frank-archon` since 2026-10-08 04:09
restarted the **staging worktree**, not the deploy checkout — a campaign
drop-in `frank-primary.service.d/bench.conf` overrode `Environment=` and
drop-in `frank-archon.service.d/bench.conf` overrode `Environment=` and
`ExecStart`. The health gate went green each time. Verified parity before
removing it (serve script + unit byte-identical, env identical except the
alias, staging HEAD an ancestor of `main`), moved the drop-in to
`/home/dank/.AGENTTRASH/20261009-frank-primary-bench-dropin/`, and
`/home/dank/.AGENTTRASH/20261009-frank-archon-bench-dropin/`, and
restarted: `/v1/models` then answered `frank-archon`. Post-deploy
behavioural proof is now part of the runbook.
- Repo: seams re-pinned (`frank_secondary`/`frank_large`/`frank_primary`
- Repo: seams re-pinned (`frank_zealot`/`frank_dragoon`/`frank_archon`
unit bats), fleet harnesses retargeted (`bench-probe`, `deep-probe`,
`calib-push`, `fleet-probes-swift15`, `autoresearch*`,
`bench-threads-batch-http`, `bench-decode`), `shared/homepage/docs.html`
@@ -75,7 +95,7 @@
anywhere; rotation is Frank's call.
## 2026-10-08
- **frank-primary (Dankserver): Swift1.5-Qwen3.8-Flash-Next GPTQ w4 —
- **frank-archon (Dankserver): Swift1.5-Qwen3.8-Flash-Next GPTQ w4 —
the radiance quantization campaign.** ukisai's BF16 checkpoint (335.3
GiB, 397/397 sha256-verified) → `qwen4exp-swift15-w4dn8-i8-hc8m.rad`
(~114 GiB): all 49,152 experts GPTQ w4 (act-order, damp 0.05) calibrated
@@ -115,10 +135,10 @@
levels x4, reasoning_content, response_format, vision — all PASS.
KLD gate vs the BF16 reference: recorded on 1.0.13 (50,235 positions,
197 prompts), scored on the same pin at the w4-legal shape.
Verdict: see runbook [[frank-primary-radiance-swift15-w4dn8]].
Verdict: see runbook [[frank-archon-radiance-swift15-w4dn8]].
## 2026-10-06
- **frank-secondary (Dankclaw): ubatch 2048 — the second rung (llama.cpp,
- **frank-zealot (Dankclaw): ubatch 2048 — the second rung (llama.cpp,
RTX 3090).** "set ubatch to 2k" → commit d8da7a9: `UBATCH_SIZE`
1024 → 2048 at the 2×114,688 shape. **Cost 746 MiB**, not the 616 MiB
the same rung measured on the MiMo shape on 2026-10-05 (a rung's price
@@ -135,16 +155,16 @@
4096, so the seam's batch ≥ ubatch guard holds; **2048 is the ceiling
for this shape** (4096 = no micro-split doubles the buffer growth
again against 0.7–1.0 GiB of headroom). e2e gate 33/33 green.
- **frank-secondary (Dankclaw): RENAMED from frank-small, then reshaped
- **frank-zealot (Dankclaw): RENAMED from frank-small, then reshaped
to 2×114,688 / ubatch 1024 (llama.cpp, RTX 3090).** The rename (commit
87721ab, 87 files: serve trio, unit, bats seam, 9 runbooks, identifiers
`FRANK_SECONDARY_*`/`llm_secondary`/`deploy_llm_secondary`; historical
`FRANK_ZEALOT_*`/`llm_zealot`/`deploy_llm_zealot`; historical
refs to the OLD FreeToken/CUDA frank-small and to the Dankserver
service that became frank-medium stay frank-small, as does
`freetoken-frank-small.md`). Live unit transition by hand before the
deploy: `systemctl stop frank-small` + `systemctl disable frank-small`
(a relinked unit — disable removes only its stale symlink) →
`deploy.sh dankclaw --service frank-secondary`. The reshape (commit
`deploy.sh dankclaw --service frank-zealot`. The reshape (commit
f2575a1, Frank's order "lower max context to 112k so we can increase
ubatch to at least 1k"): `CTX_SIZE 262144→229376` (per-slot
131,072→114,688), `UBATCH_SIZE 512→1024` → **22,374 used / 1,751 MiB
@@ -154,8 +174,8 @@
boot's 63.3/72.9 is confounded by prompt text, not a shape verdict).
Harness retargeted: models.yml (id/name/compactionModels,
`contextWindow: 114688`) + config.yml modelRoles ×14. Runbook renamed:
`serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md`.
- **frank-secondary (Dankclaw): two Swift-1.5-27B reshapes (llama.cpp, RTX 3090).**
`serverAppConfigs-docs/LLM/frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688.md`.
- **frank-zealot (Dankclaw): two Swift-1.5-27B reshapes (llama.cpp, RTX 3090).**
Morning: Frank's order "drop mtp, skip ngram, ubatch 512, two 128k
contexts" → 2×131,072 no-spec on the IQ3_S-mtp (commit 812d9eb):
22,728 used / 1,397 MiB free (the tight fit he ordered; the drafter's
@@ -172,14 +192,14 @@
uncensored model thinks more — 64 tokens is all `reasoning_content`).
ngram-mod still skipped (recurrent-state rollback law on the qwen35
hybrid). Runbook renamed + rewritten:
`serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md`.
- **frank-large (Dankclaw): halogen 27B-dense → halogen-flash-server
`serverAppConfigs-docs/LLM/frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688.md`.
- **frank-dragoon (Dankclaw): halogen 27B-dense → halogen-flash-server
0.16.2 v2 Swift1.5 ABLITERATED (Strix Halo iGPU, port 7770).** Cutover
commit 31b1ec3 (03:07:40): the v2 `.hgn` payload loads ONLY in the
flash-server line, so the engine moved with the model. Abliterated base
(refusals gone), vision back (the v2 tower loads beside the checkpoint),
MTP head inside the checkpoint, 2×200,192 slots, `HALOGEN_SPEC_ADAPT=0`.
Runbook: `serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md`.
Runbook: `serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-swift15.md`.
**Incident (03:09:39–03:11:42):** the deploy started the unit while host
`MemAvailable` was ~74.7 GiB, 3.34 GiB short of the engine's pin-gate
requirement (62.11 GiB checkpoint + 16 GiB floor = 78.11 GiB); it
@@ -187,16 +207,16 @@
self-resolved in ~2 min when the dying 27B engine freed RAM (boot #9
pinned clean at 88.4 GiB available). **Latent risk + fix:** the host was
left with only ~1.8 GiB `MemAvailable`, so a restart would crash-loop
again; `serve-frank-large.sh` now has a host-RAM preflight that checks
again; `serve-frank-dragoon.sh` now has a host-RAM preflight that checks
`MemAvailable` against the pin gate BEFORE `podman run` and WAITS up to
`FRANK_LARGE_RAM_WAIT` (600 s) for the RAM to clear, refusing loudly if
`FRANK_DRAGOON_RAM_WAIT` (600 s) for the RAM to clear, refusing loudly if
it never does (the 16 GiB floor is the engine's, not
`HALOGEN_HOST_RESERVE_GIB`).
## 2026-10-05
- **frank-primary: tcclaviger vLLM fork → FreeToken (PR #587 ROCm) on
- **frank-archon: tcclaviger vLLM fork → FreeToken (PR #587 ROCm) on
dankserver.** Built FreeToken at PR head `77a7201` on the host into the
local image `frank-primary-freetoken:pr587-77a7201`
local image `frank-archon-freetoken:pr587-77a7201`
(digest-pinned rocm/pytorch rocm7.14 torch-2.11.0 base; two build laws
bought with failures: the pip-ROCm wheel needs a `libhiprtc.so` dev
symlink for the PR's `_cpu_moe` link, and a stale `docker login` at
@@ -206,7 +226,7 @@
299,968 tokens (multiple-of-64 — the `qsa_sparse` auto backend overrides
page size to 64; 300,000 was refused), 2 slots, parsers AUTO (thinking →
`reasoning_content`; models.yml flipped accordingly). Container renamed
to the unit name (Frank's call; `frank-primary-qwen` → legacy list);
to the unit name (Frank's call; `frank-archon-qwen` → legacy list);
deployer image gate → `podman image exists` (local builds carry no
RepoDigests) and health gate reads the BODY — FreeToken 200s with
`{"status":"loading"}` mid-load, which false-greened a deploy at 6 s
@@ -217,8 +237,8 @@
single-card LRU; TP=2 exists but the pynccl JIT can't compile for
gfx1201 — probed, signature in the runbook). `/metrics` is a 404 — the
Dankclaw Prometheus llm-fleet scrape of :7770 reads down until
repointed. Rollback: tag `frank-primary-vllm-shapeG-20261003`.
Runbook: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`.
repointed. Rollback: tag `frank-archon-vllm-shapeG-20261003`.
Runbook: `serverAppConfigs-docs/LLM/frank-archon-freetoken-pr587.md`.
- **Skill library distilled into the vault (waves 1–7).** All ~173 managed
skills read in full and merged, one polished reference page per
skill-cluster:
@@ -231,7 +251,7 @@
Triage, FreeToken Builds, plus deltas folded into the retired runbooks
(GLM DFlash2 port laws, Lucebox dual-R9700, R9V qualification pipeline,
FLM pack registration, ik hybrid envelope, frank-medium split
rebalance, frank-primary llama.cpp GGUF preflight, batch-ubatch split
rebalance, frank-archon llama.cpp GGUF preflight, batch-ubatch split
section, bench-window D-state/stale-read notes, second-endpoint
bring-up in Model Swap).
- **Architecture (3):** Stale Signal Audit, Quadlet Lifecycle, Deploy
@@ -273,12 +293,12 @@
Live pass of the frankbook walk itself is deferred to the next deploy with
an awake host.
- **Catch while verifying: the 17:47 Ornith-1.5-9B swap (`77decb5`) had
left `frank_secondary_unit.bats` and the deployer's payload gate stale** —
left `frank_zealot_unit.bats` and the deployer's payload gate stale** —
the seam still expected MiMo-Q6K names and `deploy-remote.sh` gated
`LLM_SMALL_MODEL_FILE`/`LLM_SMALL_MMPROJ` on files the unit no longer
loads (the anti-drift seam doing its job). Both repointed to the
Ornith-Q8 paths; full `./tests/run.sh` back to green (366 tests).
- **frank-secondary: compute-buffer pair doubled to 4096/2048, and the prefill win
- **frank-zealot: compute-buffer pair doubled to 4096/2048, and the prefill win
measured rather than assumed.** Frank asked for the batch and ubatch doubled.
The blocker was on record: the Q5_K_M predecessor documented **"ubatch 2048
failed the scratch boot"**, and this shape carries 616 MiB less free space
@@ -288,7 +308,7 @@
`FRANK_SMALL_ENV_FILE` systemd drop-in, and it reproduced the 2026-10-03
boot **byte-identically** (21,276 used / 2,849 free), which is what makes
ubatch the only variable. Repo never saw a fake revert commit; scaffolding
staged to `~/.AGENTTRASH/20261005-frank-secondary-ubatch-ab/`.
staged to `~/.AGENTTRASH/20261005-frank-zealot-ubatch-ab/`.
- **At 8,928 prompt tokens: prefill 3,972/4,077 -> 4,207/4,212 t/s, +4.6 %,
for 616 MiB (free 2,849 -> 2,279).** Decode flat (91.9–93.0 vs 93.8–94.3 —
noise). The control that proves it is geometry and not drift: a 279-token
@@ -311,10 +331,10 @@
edge of the 2–3 GiB floor. Deployed `e5893dc`, e2e gate PASSED (2 mini specs
flaky-then-passed once the sleeping FrankBook-Pro woke mid-run — WoL took a
second burst and ~90 s). Docs:
[[frank-secondary-llama-mimo-q6k-2x200704]] (*Compute-buffer geometry*).
[[frank-zealot-llama-mimo-q6k-2x200704]] (*Compute-buffer geometry*).
## 2026-10-03
- **frank-primary: shape F then shape G — MNBT 2048, the CUDA-graph plan term
- **frank-archon: shape F then shape G — MNBT 2048, the CUDA-graph plan term
pinned to the measured capture, KV forced to 400,700 tokens, and UTIL 1.0
proven UNBOOTABLE.** Frank asked what the batch was with GPUs showing open
space; the audit found the arena is `budget - non-expert - KV reserve -
@@ -351,7 +371,7 @@
Runbook: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
## 2026-10-01
- **frank-primary rolled BACK to Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4;
- **frank-archon rolled BACK to Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4;
GLM/DFlash2 declared failed and purged.** The DFlash2 drafter booted and
served (window 6j, 65,536 ctx, `glm53-flash` on :7773, both TP ranks
captured FULL dflash2 cuda graphs — five patches deep: EAGLE3 wrapper
@@ -370,7 +390,7 @@
unifier/tailpin/grouping/registry fixes, apply-verified).
## 2026-09-30
- **frank-primary cut over to GLM-5.3-Flash W4A16 int4 — it serves, but NOT at
- **frank-archon cut over to GLM-5.3-Flash W4A16 int4 — it serves, but NOT at
the 35 tok/s target, and the shortfall is measured rather than inferred.**
- Payload `Intel/GLM-5.3-Flash-W4A16-AutoRound` (AutoRound symmetric int4
group 128, `auto_round:auto_gptq` packing, ~170 GiB, 34 shards + 2 extras,
@@ -498,10 +518,10 @@
## 2026-09-29
- **frank-large tuning investigation: a measured NEGATIVE result, nothing
- **frank-dragoon tuning investigation: a measured NEGATIVE result, nothing
changed.** Frank asked for iterative decode/prefill improvement at a 32k
working point without touching the model or quantization. **No win exists
at gufo's served settings**, so no config change was shipped. frank-large
at gufo's served settings**, so no config change was shipped. frank-dragoon
was stopped for the whole investigation (~4h) per the benchmarking policy
and restored healthy in 9s afterwards.
- Measured on the repo's own prose and shell sources at **32,762 prompt
@@ -631,7 +651,7 @@
script; the 256k config relaunched with it. Host-side checks: health
200 on 127.0.0.1:7773 AND 192.168.0.235:7773, /v1/models lists
qwen3.8-flash-next, decode through the published path
36.49/34.55/35.98 (same band). 7773 chosen over 7770 — frank-primary
36.49/34.55/35.98 (same band). 7773 chosen over 7770 — frank-archon
owns 7770 when it ever returns.
## 2026-09-28
@@ -648,7 +668,7 @@
- **The 29.3 G rollback target went with them**, by Frank's decision. The
payload-only rollback is now a re-fetch first; both the runbook and
AGENTS.md say so. **The DFlash2 drafter was NOT reclaimed** — only the
`.gguf` was taken out of that directory, because the live frank-large
`.gguf` was taken out of that directory, because the live frank-dragoon
drafter lives in it. The manifest warns against restoring the directory
wholesale.
- **Kept deliberately:** `checkpoints/` 23 G. It is unreferenced by the
@@ -657,28 +677,28 @@
**sparse** file, 3.3 G apparent but 4 K real, so removing it frees
nothing.
- **A live reference check came first, and it earned its keep.**
`serve-frank-secondary.sh`'s *fallback* `MODEL_DIR` still pointed at the
retired GSQ-RCO payload frank-secondary served on 2026-09-16/17. The env
`serve-frank-zealot.sh`'s *fallback* `MODEL_DIR` still pointed at the
retired GSQ-RCO payload frank-zealot served on 2026-09-16/17. The env
file has always overridden it, so the drift was invisible — but a
missing or unreadable env file would have turned a config problem into a
"missing asset" naming a model that no longer exists. Repointed at
`Gemma4-12B-QAT/`, which is what the env file already sets. Every other
hit was a comment, a retired fetch script's destination, or
`retire_halogen_large_quadlet` (which moves a *quadlet*, not a model).
`retire_halogen_dragoon_quadlet` (which moves a *quadlet*, not a model).
- **Not taken, by Frank's decision — scope was models only:** podman
images (**279 G reclaimable, 95%**; 170 images, 23 active) and the
uv+pip caches (**73 G**). Together that is 352 G of regenerable garbage
larger than the models reclaimed, and it is the obvious next move.
- Pre-existing: **118 G of trash from 2026-09-21** was already staged
(`20260921-frank-large-assembled` 88 G, `-redownload` 30 G) and never
(`20260921-frank-dragoon-assembled` 88 G, `-redownload` 30 G) and never
purged. Total now staged on this host: **627 G**.
- Verified after the moves: frank-large and frank-secondary both `active`,
- Verified after the moves: frank-dragoon and frank-zealot both `active`,
both `/health` 200, and all three live assets (Swift target, DFlash2
drafter, Gemma4 target) still present.
- **frank-large payload swapped to Swift-1.5 (uncensored)** (production
- **frank-dragoon payload swapped to Swift-1.5 (uncensored)** (production
change, Dankclaw). Frank asked whether the UD Q8 of
`ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` would load on
gufo, then said to do it. Port 7770, alias `frank-large`, one 262,144
gufo, then said to do it. Port 7770, alias `frank-dragoon`, one 262,144
session, the DFlash2 drafter and the client contract are unchanged; the
27B dense weights are an **abliterated (uncensored)** fine-tune of the
same base, under the **non-OSI SWIFT OPEN LICENSE 1.0**.
@@ -696,7 +716,7 @@
which both files carry as `4` and which agrees with it.
- **Fetched and verified:** 31,457,990,272 bytes, sha256 `9076f99a…`
matching the Hub LFS oid, via the new
`/scripts/download-frank-large-swift.sh`. It sends **no auth header** —
`/scripts/download-frank-dragoon-swift.sh`. It sends **no auth header** —
the repo is public and ungated, and a token on the curl command line is
world-readable via `/proc/<pid>/cmdline`. The anonymous transfer ran at
~1.8 MiB/s; resuming with the host's token through a **mode-600
@@ -730,9 +750,9 @@
tunable. That broke the deployer's stray-run sweep, which keyed on the
*target* filename and would have silently stopped matching — a stray
run would then have held 7770 past the port gate. It now keys on the
drafter, which every frank-large command line carries via
drafter, which every frank-dragoon command line carries via
`--dflash-model` and which did not move.
- **`/tests/frank_large_unit.bats`** — 20 assertions. The mutation pass
- **`/tests/frank_dragoon_unit.bats`** — 20 assertions. The mutation pass
found **four real gaps** on the first run, all now fixed: two fetch
assertions were satisfied by a **header comment** rather than the URL
assignment, so repointing a repo left them green; and nothing bound the
@@ -745,12 +765,12 @@
the `1..N` count before believing a result. 16/16 mutations caught.
- Docs: runbook gained a **Payload** section (the header evidence),
**The template law** section and a two-way **Rollback**; AGENTS.md
fleet table, frank-large section and rollback.
- **frank-large runtime swapped to gufo** (production change, Dankclaw).
fleet table, frank-dragoon section and rollback.
- **frank-dragoon runtime swapped to gufo** (production change, Dankclaw).
Frank asked for `github.com/gufo-org/gufo` — a vertical inference engine
built for exactly this hardware (Ryzen AI MAX+ 395 / Radeon 8060S,
`gfx1151`) — in place of the stock llama.cpp prebuilt. Port 7770, alias
`frank-large`, one 262,144-token session and the client contract were all
`frank-dragoon`, one 262,144-token session and the client contract were all
unchanged; only the engine was new, and the payload was the unsloth
Qwen3.8-27B UD-Q8_K_XL **at that point** — it was swapped for the
abliterated Swift-1.5 later the same day, see the entry above.
@@ -791,17 +811,17 @@
RSS 30.6 GiB; a live short-prompt request returned
`Server-Timing: ttft;dur=549.8, inter_token;dur=24.2` (~41 tok/s).
Verified live: `/health` 200, `/ready` 200, `/v1/models` →
`frank-large` at `context_length 262144`, a chat completion carrying
`frank-dragoon` at `context_length 262144`, a chat completion carrying
`reasoning_content`, a `tool_calls` response with
`finish_reason: tool_calls`, and SSE streaming.
- **The trade, stated plainly.** gufo's own Q8_K_XL tables against
llama.cpp b11069: prefill +37 % to +80 %, cold load 4.5× faster, decode
+7 % at zero depth — but **−14 % at 65,536 tokens of depth and −24 % at
131,072**. On the 8-bit target the crossover is ~32k and frank-large runs
131,072**. On the 8-bit target the crossover is ~32k and frank-dragoon runs
one 262,144-token conversation, so long agent sessions sit on the losing
side of it. The Q4_K_XL target shows no such regression; revisiting the
quantization is a separate decision.
- `/tests/frank_large_unit.bats` — Rewritten around behaviour: the seam
- `/tests/frank_dragoon_unit.bats` — Rewritten around behaviour: the seam
runs the serve script against a stub engine and asserts on the argv it
actually execs, instead of grepping the script's text. A caveat found
while writing it: **bats 1.13 ignores a failing `A && B` in the middle of
@@ -810,18 +830,18 @@
Mutations confirmed the seam bites: dead runtime gate, `DRAFT_TOKENS=0`,
a wrong context/sessions/port, an alias swap, a moved drafter, a removed
missing-asset message and a neutered commit pin each fail it.
- `/scripts/deploy-remote.sh` — `deploy_llm_large` gates on the runtime
- `/scripts/deploy-remote.sh` — `deploy_llm_dragoon` gates on the runtime
binary as well as the payload and returns **before** it stops the serving
unit, so a host whose gufo was never built reports the failure instead of
taking the endpoint down on the way there.
- Docs: new `/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md`;
`frank-large-unsloth-qwen38-27b-q8xl.md` marked RETIRED (still the
- Docs: new `/serverAppConfigs-docs/LLM/frank-dragoon-gufo-qwen38-27b.md`;
`frank-dragoon-unsloth-qwen38-27b-q8xl.md` marked RETIRED (still the
rollback target — the prebuilt and the payload are both intact on disk);
AGENTS.md fleet table, frank-large section, bench policy and runbook
AGENTS.md fleet table, frank-dragoon section, bench policy and runbook
index; both vault indexes; `Dankclaw System Specs.md`; `README.md`.
- **e2e gate: 32/33.** The one failure is `192.168.0.112` (the FrankBook-Pro
Mac mini) at 100 % packet loss — offline, and the same failure recorded
in `564a781`. Not related to this change; no frank-large test failed.
in `564a781`. Not related to this change; no frank-dragoon test failed.
- **Frontier-MoE feasibility study on Dankserver** (research + build, no
production change). Frank asked which of DeepSeek-V4.1-Flash,
MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash would run best on the 2x R9700 box,
@@ -855,71 +875,71 @@
- `/serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md` — new
runbook; indexed in `LLM/index.md`.
- **LANDMINE, not changed here:** `/home/dank/models` was cleared by Frank
mid-session to free space. frank-primary is still answering `:7770` from
mid-session to free space. frank-archon is still answering `:7770` from
mmapped-but-unlinked inodes (`/models` is empty inside the running
container) — **the endpoint cannot be restarted until the 175 GiB payload
is re-fetched.** A reboot or a container restart takes frank-primary down.
is re-fetched.** A reboot or a container restart takes frank-archon down.
- Deploy-state reconciliation (Frank's call: no restart) — the autoresearch trio
(`f187165` PLE 1800, `41e9189` prefix caching, `ba2100e` docs) was found already
MERGED to main + pushed AND already live on frank-primary: the running
`frank-primary-r9v` container (booted 12:13 from the `ba2100e` checkout) carries
MERGED to main + pushed AND already live on frank-archon: the running
`frank-archon-r9v` container (booted 12:13 from the `ba2100e` checkout) carries
`--enable-prefix-caching` in argv and `VLLM_PLE_OFFLOAD_READY_TIMEOUT=1800` in env,
`/health` 200 — but `deploy-state/frank-primary` was still stamped `f187165`. A
`deploy.sh` run would have re-selected frank-primary and reloaded the 175 GiB
`/health` 200 — but `deploy-state/frank-archon` was still stamped `f187165`. A
`deploy.sh` run would have re-selected frank-archon and reloaded the 175 GiB
payload (~16–19 min) for zero behavioral change, on the endpoint this harness itself
runs on. Stamped the marker `f187165 → ba2100e` (status `deployed`, atomic
mktemp+mv per the deployer idiom); service untouched. e2e gate 32/33 — the lone
failure `11-admin` is the FrankBook-Pro Mac mini (`192.168.0.112:8989`) offline
(100 % ICMP loss + TCP timeout), the documented asleep-host condition, not a
regression.
- `scripts/frank-primary.env`, runbook — **prefix caching DEPLOYED on
frank-primary** (Frank's go-ahead; commit `41e9189`): E8-qualified, zero
- `scripts/frank-archon.env`, runbook — **prefix caching DEPLOYED on
frank-archon** (Frank's go-ahead; commit `41e9189`): E8-qualified, zero
VRAM cost, 9x/63x reuse TTFT. Verified on :7770 (argv flag, health 200,
completion "PREFIX-CACHE-LIVE"). Gotcha re-confirmed: the container is
in dank's ROOTLESS podman store (`sudo podman` inspects an empty store
and reports nothing); probe via the shim's `docker` or the unit MainPID.
- `/scripts/frank-primary.env`, `/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md`, `/serverAppConfigs-docs/index.md` — **frank-primary prefill/decode bench matrix** (scratch :7773, prod stopped 00:06 for the window per fleet bench law): baseline E0 (chunk 4096 / 80-slot sync / MTP4) = PP8K 954.6, PP131K 883.0 tok/s, decode 141.9 ms/step ≈ 23.8 t/s effective. Chunk 8192 (E1R): PP8K +25.9% (1201.4) but decode −8.5% and PP131K −4.4% — short-prompt win, decode tax, not the fleet's dominant shape. Chunk 16384 boots but OOMs at request time (0.93 GiB free) — 4096 is the ceiling (engine spec-config cap + memory). Expert cache 80 slots is the policy max (doctor: capacities 48/64/80, synchronous LRU only — async unsupported, 96/112 rejected); MTP depth 6 is 3.7x SLOWER than depth 4. Prefix caching (E8/E9): zero VRAM cost, cold path unchanged, reuse hits 32k 44.2→3.6–4.3 s (9x), 131k 439→6.9 s (63x); hit pattern rounds 1–2 miss / 3+ hit — RECOMMENDED, pending Frank's go-ahead. Two failure laws recorded: unqualified shapes fail mid-request (health stays 200), and PLE registration can blow the engine's 600 s `VLLM_PLE_OFFLOAD_READY_TIMEOUT` under page-cache churn (measured 740 s) — launch.sh gained `R9V_PLE_READY_TIMEOUT` forwarding (fork `2de0357`, local-only) and prod now sets 1800.
- **Incident — frank-primary down 00:06 → ~11:00 (11h)**: the bench window's required prod stop was never restored — the matrix + follow-ups ran 00:44 → 09:02 and the session let the endpoint sit dead (the 2026-09-18 "a bench session owes the fleet a health 200" lesson, re-offended). The first restore boot (09:46) then failed on the PLE 600 s timeout above (unit "Deactivated successfully", no journal error — the serve script exits 0 on the wait). Restored by the `R9V_PLE_READY_TIMEOUT=1800` fix deploy (f187165, from main — the deploy gate refuses non-main): health 200 + MainPID = the serve script's own `podman wait` + live completion verified ~11:00. Deploy e2e gate 32/33 — the one failure (11-admin) needs the FrankBook-Pro Mac mini, which was asleep and ignored two WoL bursts (240 packets); recurring pattern per the wol-wake skill. Lesson: a multi-hour bench window needs an explicit restore task, not a "watch for DONE" — DONE ≠ prod up.
- `/scripts/frank-archon.env`, `/serverAppConfigs-docs/LLM/frank-archon-r9v-radiance-qwen38-q8.md`, `/serverAppConfigs-docs/index.md` — **frank-archon prefill/decode bench matrix** (scratch :7773, prod stopped 00:06 for the window per fleet bench law): baseline E0 (chunk 4096 / 80-slot sync / MTP4) = PP8K 954.6, PP131K 883.0 tok/s, decode 141.9 ms/step ≈ 23.8 t/s effective. Chunk 8192 (E1R): PP8K +25.9% (1201.4) but decode −8.5% and PP131K −4.4% — short-prompt win, decode tax, not the fleet's dominant shape. Chunk 16384 boots but OOMs at request time (0.93 GiB free) — 4096 is the ceiling (engine spec-config cap + memory). Expert cache 80 slots is the policy max (doctor: capacities 48/64/80, synchronous LRU only — async unsupported, 96/112 rejected); MTP depth 6 is 3.7x SLOWER than depth 4. Prefix caching (E8/E9): zero VRAM cost, cold path unchanged, reuse hits 32k 44.2→3.6–4.3 s (9x), 131k 439→6.9 s (63x); hit pattern rounds 1–2 miss / 3+ hit — RECOMMENDED, pending Frank's go-ahead. Two failure laws recorded: unqualified shapes fail mid-request (health stays 200), and PLE registration can blow the engine's 600 s `VLLM_PLE_OFFLOAD_READY_TIMEOUT` under page-cache churn (measured 740 s) — launch.sh gained `R9V_PLE_READY_TIMEOUT` forwarding (fork `2de0357`, local-only) and prod now sets 1800.
- **Incident — frank-archon down 00:06 → ~11:00 (11h)**: the bench window's required prod stop was never restored — the matrix + follow-ups ran 00:44 → 09:02 and the session let the endpoint sit dead (the 2026-09-18 "a bench session owes the fleet a health 200" lesson, re-offended). The first restore boot (09:46) then failed on the PLE 600 s timeout above (unit "Deactivated successfully", no journal error — the serve script exits 0 on the wait). Restored by the `R9V_PLE_READY_TIMEOUT=1800` fix deploy (f187165, from main — the deploy gate refuses non-main): health 200 + MainPID = the serve script's own `podman wait` + live completion verified ~11:00. Deploy e2e gate 32/33 — the one failure (11-admin) needs the FrankBook-Pro Mac mini, which was asleep and ignored two WoL bursts (240 packets); recurring pattern per the wol-wake skill. Lesson: a multi-hour bench window needs an explicit restore task, not a "watch for DONE" — DONE ≠ prod up.
## 2026-09-27
- `/scripts/frank-primary.env`, `/scripts/frank-primary.service`, `/scripts/serve-frank-primary.sh`, `/scripts/deploy-remote.sh`, `/tests/frank_primary_unit.bats`, `/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md` (new), AGENTS.md + LLM index — **frank-primary cut over to the Q8_0 payload** (uncensored IQ4_XS → unsloth Qwen3.8-Flash-Next Q8_0, 175.30 GiB / 6 shards) at Frank's direction. Launcher gained the opt-in `R9V_CHAT_TEMPLATE_FILE`/`R9V_CHAT_TEMPLATE_KWARGS` passthrough (fork commit `30d4a32`, local-only) — the Qwen-Sharp v22.4.0 template now serves, fleet parity with frank-large. Env points at `/home/dank/r9v-state/q8xl/setup.json` + the quantized-inject dev-overlay adapter (fork `363d606`) + `R9V_FULL_MUTABLE_CACHE=0`; shard gates extended to SHARD4/5/6 (unset-safe); seams recalibrated; ctx window 131072 (models.yml maxTokens 36864). Spec decode is MTP4 only — the fork's `speculative_config` is a single dict, no chaining, so llama.cpp's `draft-mtp,ngram-mod` pattern is NOT expressible (PLE n-gram rides along architecturally). Uncensored state + payload remain on disk as the rollback target. 346 tests green.
- Dankserver frank-primary slot (experiment window) — the `radiance-vllm-qwen4exp` fork qualified the unsloth `Qwen3.8-Flash-Next-GGUF` **Q8_0** payload (175.30 GiB, 6 split parts) on the dual R9700. Two launch blockers fixed loader-side in the fork: `runtimes/dev-overrides/qwen4_exp.py` dequantizes quantized `block_inject` shards in-stream (the Q8_0 payload quantizes BOTH fused-HC shards; IQ4_XS ships inject as F32 — the uniformity check rejected the mixed projection) and `R9V_FULL_MUTABLE_CACHE=0` passthrough for the 80-slot tiered cache (fork commits 363d606, 2fe6dbb, 50a79ec — GitHub remote, not pushed). Battery on :7773: text + reasoning channel, vision ("Red" on a red PNG), prose ≈11.0 tok/s wall, MTP4 mean acceptance length 3.19–5.00; PLE pinned residency proven (`mapping_bytes=54400261120`, `mapping_rss_bytes` == mapping size, both ranks required 108.8 GiB vs 140.9 available). Relaunch recipe saved at `dankserver:/home/dank/r9v-state/q8xl/START-COMMAND.txt`; record at fork `docs/qualification/qwen38-q8-0-dual-r9700-2026-09-27.md`.
- Dankserver frank-primary restore — uncensored UD-IQ4_XS back on `0.0.0.0:7770` as `frank-primary-r9v` from its frozen config (health 200 in ~60 s; LAN-path completion returns content + reasoning); `frank-primary.service` (llama.cpp) **runtime-masked** for the window (`/run/systemd/system/frank-primary.service → /dev/null` — the `/etc` relink symlink blocks a plain mask; self-clears at reboot, unit file untouched) and the container runs `restart=no`, so a reboot returns the repo-configured llama.cpp with no contention. Restore config preserved at `dankserver:/home/dank/r9v-state/uncensored/restore-config-20260927.env`; scratch in `dankserver:~/.AGENTTRASH/20260927-q8xl-bringup/`.
- `/scripts/frank-archon.env`, `/scripts/frank-archon.service`, `/scripts/serve-frank-archon.sh`, `/scripts/deploy-remote.sh`, `/tests/frank_archon_unit.bats`, `/serverAppConfigs-docs/LLM/frank-archon-r9v-radiance-qwen38-q8.md` (new), AGENTS.md + LLM index — **frank-archon cut over to the Q8_0 payload** (uncensored IQ4_XS → unsloth Qwen3.8-Flash-Next Q8_0, 175.30 GiB / 6 shards) at Frank's direction. Launcher gained the opt-in `R9V_CHAT_TEMPLATE_FILE`/`R9V_CHAT_TEMPLATE_KWARGS` passthrough (fork commit `30d4a32`, local-only) — the Qwen-Sharp v22.4.0 template now serves, fleet parity with frank-dragoon. Env points at `/home/dank/r9v-state/q8xl/setup.json` + the quantized-inject dev-overlay adapter (fork `363d606`) + `R9V_FULL_MUTABLE_CACHE=0`; shard gates extended to SHARD4/5/6 (unset-safe); seams recalibrated; ctx window 131072 (models.yml maxTokens 36864). Spec decode is MTP4 only — the fork's `speculative_config` is a single dict, no chaining, so llama.cpp's `draft-mtp,ngram-mod` pattern is NOT expressible (PLE n-gram rides along architecturally). Uncensored state + payload remain on disk as the rollback target. 346 tests green.
- Dankserver frank-archon slot (experiment window) — the `radiance-vllm-qwen4exp` fork qualified the unsloth `Qwen3.8-Flash-Next-GGUF` **Q8_0** payload (175.30 GiB, 6 split parts) on the dual R9700. Two launch blockers fixed loader-side in the fork: `runtimes/dev-overrides/qwen4_exp.py` dequantizes quantized `block_inject` shards in-stream (the Q8_0 payload quantizes BOTH fused-HC shards; IQ4_XS ships inject as F32 — the uniformity check rejected the mixed projection) and `R9V_FULL_MUTABLE_CACHE=0` passthrough for the 80-slot tiered cache (fork commits 363d606, 2fe6dbb, 50a79ec — GitHub remote, not pushed). Battery on :7773: text + reasoning channel, vision ("Red" on a red PNG), prose ≈11.0 tok/s wall, MTP4 mean acceptance length 3.19–5.00; PLE pinned residency proven (`mapping_bytes=54400261120`, `mapping_rss_bytes` == mapping size, both ranks required 108.8 GiB vs 140.9 available). Relaunch recipe saved at `dankserver:/home/dank/r9v-state/q8xl/START-COMMAND.txt`; record at fork `docs/qualification/qwen38-q8-0-dual-r9700-2026-09-27.md`.
- Dankserver frank-archon restore — uncensored UD-IQ4_XS back on `0.0.0.0:7770` as `frank-archon-r9v` from its frozen config (health 200 in ~60 s; LAN-path completion returns content + reasoning); `frank-archon.service` (llama.cpp) **runtime-masked** for the window (`/run/systemd/system/frank-archon.service → /dev/null` — the `/etc` relink symlink blocks a plain mask; self-clears at reboot, unit file untouched) and the container runs `restart=no`, so a reboot returns the repo-configured llama.cpp with no contention. Restore config preserved at `dankserver:/home/dank/r9v-state/uncensored/restore-config-20260927.env`; scratch in `dankserver:~/.AGENTTRASH/20260927-q8xl-bringup/`.
## 2026-09-25
- `/serverAppConfigs-docs/Dankserver/Jellyfin.md` — Jellyfin LDAP 401 resolved: the auth works via the modern `/Users/AuthenticateByName` endpoint (HTTP 200, alex authenticated, full plugin flow logged: service bind → search → ldapUid → user bind → token). The 401 was a test artifact — I had been probing the legacy `/Users/Auth/Password` route, which Jellyfin 12.1 removed (the `LegacyEmbyRouteRewriteMiddleware` is gone). The web UI uses the modern endpoint, so user logins are fine. No config change needed; the LDAP-Auth 23.0.0.0 plugin + authentik 2026.8.3 outpost are healthy.
- `/shared/homepage/services.json`, `/shared/homepage/docs.html` — Suggested uses added to the three LLM model descriptions (status cards + the `/docs` model guide + the "which one to pick" foot-note): frank-large → cybersecurity + debugging (use requires Frank's permission); frank-medium → the primary intelligent model (surpasses Claude Sonnet 5 per Artificial Analysis) for software engineering, sysadmin, instruction tasks; frank-secondary → the fast/cheap utility model (agentic search, RAG, research subagents, translation, roleplay, summarization). Deployed to `/srv/homepage`; bats 19/19 + non-Mac-mini e2e green (d3c8197).
- `/shared/homepage/{services.json,app.js,style.css,docs.html}` — Model cards reworked to a structured three-line shape: each LLM card now carries a `details` object (role / spec / use) rendered as a bold pale-green role, a bright-green spec line, and a dim "Use:" line (`app.js` `descHTML()` falls back to the plain `description` for non-LLM cards; new `.desc-role`/`.desc-spec`/`.desc-use` CSS + bare-mode reset). Roles made neutral (no nicknames): frank-large = deep reasoning, frank-medium = primary model, frank-secondary = fast utility model. The `/docs` model guide was also fixed — it had drifted to retired model info (frank-large listed as Qwen3.8-Flash-Next 87.3 GiB / 200k, frank-medium as FP8 ~31 GiB) — now Qwen3.8-27B UD-Q8_K_XL 31.5 GiB / 262k and MXFP4/W4A8 19.4 GiB, reordered large→medium→small. bats 30/30 + live browser render verified.
- `/shared/homepage/services.json`, `/shared/homepage/docs.html` — Suggested uses added to the three LLM model descriptions (status cards + the `/docs` model guide + the "which one to pick" foot-note): frank-dragoon → cybersecurity + debugging (use requires Frank's permission); frank-medium → the primary intelligent model (surpasses Claude Sonnet 5 per Artificial Analysis) for software engineering, sysadmin, instruction tasks; frank-zealot → the fast/cheap utility model (agentic search, RAG, research subagents, translation, roleplay, summarization). Deployed to `/srv/homepage`; bats 19/19 + non-Mac-mini e2e green (d3c8197).
- `/shared/homepage/{services.json,app.js,style.css,docs.html}` — Model cards reworked to a structured three-line shape: each LLM card now carries a `details` object (role / spec / use) rendered as a bold pale-green role, a bright-green spec line, and a dim "Use:" line (`app.js` `descHTML()` falls back to the plain `description` for non-LLM cards; new `.desc-role`/`.desc-spec`/`.desc-use` CSS + bare-mode reset). Roles made neutral (no nicknames): frank-dragoon = deep reasoning, frank-medium = primary model, frank-zealot = fast utility model. The `/docs` model guide was also fixed — it had drifted to retired model info (frank-dragoon listed as Qwen3.8-Flash-Next 87.3 GiB / 200k, frank-medium as FP8 ~31 GiB) — now Qwen3.8-27B UD-Q8_K_XL 31.5 GiB / 262k and MXFP4/W4A8 19.4 GiB, reordered large→medium→small. bats 30/30 + live browser render verified.
- Fleet-wide container update (dankclaw + dankserver) — every pullable image brought current. Pinned bumps: authentik `2026.8.0`→`2026.8.3` (server/worker/outpost), tautulli `v2.18.1-ls242`→`ls245`, calibre-web `0.6.27-ls397`→`ls402`, plex `1.43.3.10896`→`1.43.4.10903`; seerr already at `v3.4.1`. Local-build rebuilds (forced `--no-cache`, the base images had moved): caddy (xcaddy porkbun), alloy, loki, comfyui-rocm (ComfyUI re-cloned), hermes ×3 (fresh `:latest`, `23986f31`→`e1e87aec`). anubis already at latest stable `v1.27.0` (only pre-releases exist beyond). Verified: all 16 floating-tag services CURRENT (zero stale), every real container active, caddy edge serving (auth 302 / ai 200), 340 bats green. Excluded: frank-medium/radiance-vllm (custom LLM build).
- `/dankserver/{tautulli,calibre-web,plex}/quadlet/*.container`, `/dankclaw/authentik/quadlet/*.container` — version pins bumped; docs (Tautulli.md, Calibre-Web.md, Plex.md, Authentik.md + server/worker, index.md) and the dead `dankserver/compose.yaml` plex entry synced (0b1ea2e).
## 2026-09-22
- `/serverAppConfigs-docs/**` — OKF v0.1 frontmatter migration: `type`/`description`/`tags`/`timestamp` added to the 65 non-reserved pages lacking them (timestamps from each file's last git commit); `timestamp` added to `LLM/frank-medium-batch-ubatch.md` and `LLM/frank-secondary-benchmark.md`; unquoted YAML scalars (colons, `*arr`) fixed in 13 pre-existing frontmatter blocks; frontmatter stripped from the reserved `LLM/Pipeline Parallelism/index.md`; `log.md` restructured to the OKF format (date-grouped, newest first, one line per change) from 40 narrative entries.
- `/serverAppConfigs-docs/**` — Fact-check pass against scripts/AGENTS.md ground truth: 6 false "Active" banners fixed on retired runbooks, 96→114 GiB GTT in `Shared Memory Architecture Dankclaw.md`, `deployment.md` + `Architecture/Deployment Flow.md` rewritten to the Quadlet/deploy-plan.py truth, `Calibre.md` 9.9.0→9.15.0 + CVE-2025-7404 conflation (it is the Calibre-Web CVE), `Copyparty.md` v1.20.24, `RomM.md` 5.3.0 breaking change; `*arr` docs moved to a new `FrankBook-Pro/` section (git mv, link-safe); active runbook bodies written for frank-large (unsloth UD-Q8_K_XL) and frank-secondary (Gemma4-12B).
- `/serverAppConfigs-docs/**` — OKF v0.1 frontmatter migration: `type`/`description`/`tags`/`timestamp` added to the 65 non-reserved pages lacking them (timestamps from each file's last git commit); `timestamp` added to `LLM/frank-medium-batch-ubatch.md` and `LLM/frank-zealot-benchmark.md`; unquoted YAML scalars (colons, `*arr`) fixed in 13 pre-existing frontmatter blocks; frontmatter stripped from the reserved `LLM/Pipeline Parallelism/index.md`; `log.md` restructured to the OKF format (date-grouped, newest first, one line per change) from 40 narrative entries.
- `/serverAppConfigs-docs/**` — Fact-check pass against scripts/AGENTS.md ground truth: 6 false "Active" banners fixed on retired runbooks, 96→114 GiB GTT in `Shared Memory Architecture Dankclaw.md`, `deployment.md` + `Architecture/Deployment Flow.md` rewritten to the Quadlet/deploy-plan.py truth, `Calibre.md` 9.9.0→9.15.0 + CVE-2025-7404 conflation (it is the Calibre-Web CVE), `Copyparty.md` v1.20.24, `RomM.md` 5.3.0 breaking change; `*arr` docs moved to a new `FrankBook-Pro/` section (git mv, link-safe); active runbook bodies written for frank-dragoon (unsloth UD-Q8_K_XL) and frank-zealot (Gemma4-12B).
- `/serverAppConfigs-docs/Research/` — Three new research briefs (OKF): `Local TTS and Voice Cloning.md`, `AI Job Search and Resume Tools.md`, `Most Popular Self-Hosted AI Apps.md` (the 15 most popular self-hosted AI apps of Jun–Sep 2026, ranked by GitHub stars) + a self-contained dark-theme HTML report `popular-self-hosted-ai-apps.html`; upstream-version rows added to the ComfyUI/OpenWebUI/Hermes app docs (ComfyUI 0.37.0 native Qwen-Image 2.1, OpenWebUI 0.11.4 + `/api/v1/images/generations`, Hermes v0.21.4/CalVer v2026.9.21). All 336 tests pass; 136 pages, 0 unresolved wikilinks; docs-only deploy no-op.
## 2026-09-21
- `/dankserver/frank-medium/quadlet/frank-medium.container` — frank-medium cut back to the MXFP4 checkpoint (`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq`); `KV_DTYPE` became a quadlet tunable, set `auto` (unquantized bf16 KV) to test whether fp8 KV had been degrading W4A8 output quality; pool auto-sized to 486,464 tokens; vision re-verified; e2e 32/32 (a45f396).
- `/dankserver/calibre-server/quadlet/calibre-server.container` — New boot-persistent Quadlet: Calibre Content Server on 192.168.0.235:8084 (OPDS + web UI), HTTP Basic auth, LAN-only; desktop image 9.15.0, `--auth-mode=basic` pinned, internal port 8090.
- `/serverAppConfigs-docs/Dankserver/Calibre.md` — Content-server section rewritten from "not running" to the deployed shape; Calibre-Web → authentik feasibility answered (guard chain, not in-app OIDC).
- `/scripts/serve-frank-large.sh`, `/scripts/frank-large.env`, `/scripts/frank-large.service`, `/scripts/download-frank-large-unsloth.sh` — frank-large (dankclaw :7770) moved CIRU Orca → unsloth llama.cpp b11030 prebuilt serving `unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS` (87.3 GiB), 200k ctx, MTP depth 2 via explicit `-md`; the split parts ARE the payload — concatenating them trips `invalid split file name` (part 1's header advertises `file_count=3`); draft acceptance verified live 0.83–0.92.
- `/serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-flash-next.md` — New runbook; `frank-large-ciru-orca.md` marked RETIRED (rollback target); AGENTS.md + LLM index updated.
- `/scripts/serve-frank-dragoon.sh`, `/scripts/frank-dragoon.env`, `/scripts/frank-dragoon.service`, `/scripts/download-frank-dragoon-unsloth.sh` — frank-dragoon (dankclaw :7770) moved CIRU Orca → unsloth llama.cpp b11030 prebuilt serving `unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS` (87.3 GiB), 200k ctx, MTP depth 2 via explicit `-md`; the split parts ARE the payload — concatenating them trips `invalid split file name` (part 1's header advertises `file_count=3`); draft acceptance verified live 0.83–0.92.
- `/serverAppConfigs-docs/LLM/frank-dragoon-unsloth-qwen38-flash-next.md` — New runbook; `frank-dragoon-ciru-orca.md` marked RETIRED (rollback target); AGENTS.md + LLM index updated.
- `/shared/prometheus/prometheus.yml` — `node-dankclaw` target fixed to `host.containers.internal:9100` (netavark DNATs published ports only; the node-exporter's host bind was unreachable by construction — a fourth down job the 09-16 audit had not listed); `Infrastructure/Prometheus.md` targets table + `tests/observability_quadlet.bats` contract updated.
## 2026-09-20
- `/dankserver/frank-medium/quadlet/frank-medium.container` — Checkpoint swapped MXFP4 → FP8 (`orcarouter/Qwen3.8-27B-Uncensored-FP8`, MTP self-speculation depth 8, acceptance 5.66/8); the swap is now one command via `scripts/hf-manifest.py` + `scripts/hf-fetch.sh` + `scripts/swap-frank-medium.sh`; A/B on the FP8 target: DFlash2 100.0 t/s decode beats MTP 85.9 (accept length is a proxy, not the metric) — DFlash2 is the deployed shape.
- `/dankserver/frank-medium/quadlet/frank-medium.container` — Cutover to vllm-radiance TP=2 (95.8 t/s / 0.21 s TTFT / 259.4 burst-4 vs llama layer split 45.7 / 0.72 / 71.7; the winning arm is also 4-bit, part of the margin is quant); the llama root unit moved to `scripts/.AGENTTRASH/20260920-llama-layersplit/` (never `systemctl disable`); KV pool auto-sized to 943,581 tokens (3.59× the window); tools, reasoning and vision verified live against the endpoint.
- `/scripts/build-llama-rdna4.sh` — Now takes `GGML_HIP_RCCL`; the RCCL build (`build-rdna4-rocm714-rccl`) made the tensor-mode bench arm fair (Ubuntu RCCL 2.27 carries gfx1201 device code; TheRock's does not — `invalid device function`).
- `/shared/prometheus/rules/llm-fleet.yml`, `/shared/prometheus/prometheus.yml`, `/shared/grafana/provisioning/` — Monitoring spans the `vllm:` + `llamacpp:` namespaces (6 alerts + 3 recording rules); frank-secondary/frank-large targets fixed to `host.containers.internal`; the prefix-cache reuse rule divides by `cached + prompt` (llama.cpp's `prompt_tokens_total` EXCLUDES cached tokens); prefix caching measured at 1,648-token block granularity (13,021-token prompt 2.30 s → 0.68 s, 76 % reused).
- `/shared/prometheus/rules/llm-fleet.yml`, `/shared/prometheus/prometheus.yml`, `/shared/grafana/provisioning/` — Monitoring spans the `vllm:` + `llamacpp:` namespaces (6 alerts + 3 recording rules); frank-zealot/frank-dragoon targets fixed to `host.containers.internal`; the prefix-cache reuse rule divides by `cached + prompt` (llama.cpp's `prompt_tokens_total` EXCLUDES cached tokens); prefix caching measured at 1,648-token block granularity (13,021-token prompt 2.30 s → 0.68 s, 76 % reused).
- `/scripts/dankserver-route-primary.sh` — `eno1` pinned primary (metric 100), `eno2` retained failover (metric 200); applied with `nmcli device reapply` (no `connection reapply` in NM 1.54).
- Hardware — dankserver board swapped to the ASUS ProArt B850-Creator WiFi NEO: both R9700s on CPU root ports at Gen5 x8, every bridge `AtomicOpsCap: Routing+`, both endpoints `ReqEn+` — the PCIe-atomics wall is gone; GPU BDFs moved, kernel 7.0.0-31, SMT back on (16 CPUs).
- `/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` — Now the ACTIVE runbook (deployed flags, battery, capabilities, rollback); `frank-medium-llamacpp-dflash2.md` carries the SUPERSEDED banner; `Dual-R9700-Cooperation.md` states the measured verdict; `Dankserver System Specs.md` gained a Motherboard section; `frank-medium-model-swap.md` is the new swap runbook; AGENTS.md topology/sections/lessons flipped.
## 2026-09-19
- `/scripts/frank-large.service`, `/scripts/serve-frank-large.sh` — frank-large: halogen → CIRU v4.4.1 Orca (jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca w4b, 262144 ctx, MTP depth 4, 4096 MiB PLE cache, text only); halogen is the rollback target (b6ce3af, f68bda5).
- `/serverAppConfigs-docs/LLM/frank-large-ciru-orca.md` — New runbook.
- `/scripts/frank-dragoon.service`, `/scripts/serve-frank-dragoon.sh` — frank-dragoon: halogen → CIRU v4.4.1 Orca (jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca w4b, 262144 ctx, MTP depth 4, 4096 MiB PLE cache, text only); halogen is the rollback target (b6ce3af, f68bda5).
- `/serverAppConfigs-docs/LLM/frank-dragoon-ciru-orca.md` — New runbook.
- `/scripts/frank-medium.env`, `/scripts/serve-frank-medium.sh` — frank-medium: ajgazin UD-Q8_K_XL → orcarouter Q8_0 + z-lab DFlash2 Q4_K_M drafter; split re-balanced 35:30 → 32:33; pool 393216 → 327680 (d374ab9).
- `/scripts/frank-secondary.env`, `/scripts/serve-frank-secondary.sh` — frank-secondary: same-day vLLM detour reverted to mainline llama.cpp Gemma4-12B Q4_K_M + MTP sidecar + ngram-mod, q8_0 KV, 4 slots of 262144 (374041c).
- `/scripts/frank-zealot.env`, `/scripts/serve-frank-zealot.sh` — frank-zealot: same-day vLLM detour reverted to mainline llama.cpp Gemma4-12B Q4_K_M + MTP sidecar + ngram-mod, q8_0 KV, 4 slots of 262144 (374041c).
- Hardware — 3090 EVGA FTW3 iCX3 RGB set MODE_OFF and saved to EEPROM (persists across reboots, headless).
- AGENTS.md, both indexes, halogen runbook retirement banner (764cf41, 88331e2); 331 bats + 32 e2e green.
@@ -928,9 +948,9 @@
- Incident — frank-medium found stopped twice: (morning) a bench round ended without the protocol's restore half; (midday) an orphaned `llama-server` from `third_party/llamacpp/build-merge` (PPID 1, loopback :17770) held ~16 GiB on each R9700 and would have starved the 35:30 allocation — proven abandoned (frozen CPU ticks, zero connections) then killed; health 200 in ~20 s. Lesson: "GPUs idle" is a check, not an assumption.
## 2026-09-17
- `/scripts/frank-secondary.env`, `/scripts/serve-frank-secondary.sh` — frank-secondary: ISTA-DASLab GSQ-RCO IQ3_S → `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` Q4_K_M + separate MTP assistant sidecar (`--spec-type draft-mtp,ngram-mod`); q8_0 KV (`build-cuda` rebuilt with `-DGGML_CUDA_FA_ALL_QUANTS=ON`, else ~12× prefill collapse); `--fit` chose the full 262144 window; audio transcribes verbatim; Gemma4 accepts every `reasoning_effort` level.
- `/scripts/frank-secondary.env` — `CTX_SIZE=1048576` / `PARALLEL=4` (four FULL 262144-token slots, the measured card ceiling: ~3,450 MiB/slot + ~8,546 fixed); ubatch 1024 deployed (pp ~965 / tg 226–302 at 253k; ubatch 2048 OOMs the MTP draft's pp buffer); draft n-max 4 + poll 100 measured WORSE at long ctx.
- `/serverAppConfigs-docs/LLM/frank-secondary-gemma4-12b.md` — New ACTIVE runbook; `frank-secondary-qwen38-gsq-rco.md` retired as rollback target.
- `/scripts/frank-zealot.env`, `/scripts/serve-frank-zealot.sh` — frank-zealot: ISTA-DASLab GSQ-RCO IQ3_S → `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` Q4_K_M + separate MTP assistant sidecar (`--spec-type draft-mtp,ngram-mod`); q8_0 KV (`build-cuda` rebuilt with `-DGGML_CUDA_FA_ALL_QUANTS=ON`, else ~12× prefill collapse); `--fit` chose the full 262144 window; audio transcribes verbatim; Gemma4 accepts every `reasoning_effort` level.
- `/scripts/frank-zealot.env` — `CTX_SIZE=1048576` / `PARALLEL=4` (four FULL 262144-token slots, the measured card ceiling: ~3,450 MiB/slot + ~8,546 fixed); ubatch 1024 deployed (pp ~965 / tg 226–302 at 253k; ubatch 2048 OOMs the MTP draft's pp buffer); draft n-max 4 + poll 100 measured WORSE at long ctx.
- `/serverAppConfigs-docs/LLM/frank-zealot-gemma4-12b.md` — New ACTIVE runbook; `frank-zealot-qwen38-gsq-rco.md` retired as rollback target.
- `/shared/homepage/` — The three LLM cards moved to the user page with live descriptions; new static `/docs` user page (access chain, model guide, how-tos) routed in the post-auth listener (32751c0).
- `/scripts/deploy-remote.sh` — The pending pass now prunes stale `pending` state entries for retired services into `~/.AGENTTRASH` and keeps going (a stale `frank-default-sightsound` entry made every `--pending` run exit 2) (02a7876).
- Fleet audit — Every vhost classified against the guard policy (no new exposure); `systemd-hermes` logged as a boot-persistence gap (running container, no quadlet); stray stopped `test-vllm` container on dankserver held for approval.
@@ -939,10 +959,10 @@
- `/scripts/bench-batch-ubatch.sh` — New harness: one `llama-bench` invocation per cell (a test that cannot allocate `exit(1)`s, taking a whole sweep with it), per-cell VRAM sampler, service stopped/started from a trap; must run from a transient unit (`LimitMEMLOCK=infinity` is load-bearing) (87845fc, 77a07ff).
- `/scripts/frank-medium.env` — `UBATCH_SIZE` 1536 → 512 (prefill peaks at 512: 1193 vs 1078 t/s at 2048; 3.71 GiB cheaper) and `TENSOR_SPLIT` 36,29 → 35,30 (ubatch buffers cost ROCm1 ~2× what they cost ROCm0) (a11fae0); `tests/frank_medium_unit.bats` balance seam re-fitted as a shape assertion.
- `/serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md` — New page: the sweep, the logical-batch irrelevance, the amdgpu settle-after-kill trap.
- `/dankclaw/frank-large/quadlet/frank-large.container` — halogen restored as frank-large, retuned: ctx 262144 → 200192, pool 401408 → 400384, slots 3 → 2, max-tok 32768 → 24576, host-reserve 15 → 20 GiB, vision tower unset (moved to frank-secondary); ledger 95.4 GiB held / 14.7 GiB spare, no clamp.
- `/scripts/frank-secondary.env`, `/scripts/frank-secondary.service`, `/scripts/serve-frank-secondary.sh` — New frank-secondary: mainline llama.cpp 37b53fd on the RTX 3090 serving ISTA-DASLab GSQ-RCO IQ3_S with the in-file MTP head (`draft-mtp`), BF16 projector, 131072 ctx, f16 KV, port 7771.
- `/scripts/deploy-remote.sh` — `deploy_llm_large` flipped back to halogen with a `wait_port_free 7770` gate (the first cutover died on `Address already in use` two seconds after the old server exited); `deploy_llm_small` added.
- `/serverAppConfigs-docs/LLM/frank-secondary-qwen38-gsq-rco.md`, `/serverAppConfigs-docs/LLM/halogen-flash-server.md` — New / re-activated runbooks; `frank-large-ik-uncensored.md` marked retired.
- `/dankclaw/frank-dragoon/quadlet/frank-dragoon.container` — halogen restored as frank-dragoon, retuned: ctx 262144 → 200192, pool 401408 → 400384, slots 3 → 2, max-tok 32768 → 24576, host-reserve 15 → 20 GiB, vision tower unset (moved to frank-zealot); ledger 95.4 GiB held / 14.7 GiB spare, no clamp.
- `/scripts/frank-zealot.env`, `/scripts/frank-zealot.service`, `/scripts/serve-frank-zealot.sh` — New frank-zealot: mainline llama.cpp 37b53fd on the RTX 3090 serving ISTA-DASLab GSQ-RCO IQ3_S with the in-file MTP head (`draft-mtp`), BF16 projector, 131072 ctx, f16 KV, port 7771.
- `/scripts/deploy-remote.sh` — `deploy_llm_dragoon` flipped back to halogen with a `wait_port_free 7770` gate (the first cutover died on `Address already in use` two seconds after the old server exited); `deploy_llm_small` added.
- `/serverAppConfigs-docs/LLM/frank-zealot-qwen38-gsq-rco.md`, `/serverAppConfigs-docs/LLM/halogen-flash-server.md` — New / re-activated runbooks; `frank-dragoon-ik-uncensored.md` marked retired.
- `/serverAppConfigs-docs/LLM/x4-pair-experiment.md` — New page: 3090 + Strix Halo over the x4 link. iGPU-only Vulkan matches halogen's ~25 tok/s with the 3090 idle but commits 116 of 121 GiB (throughput parity, memory bankruptcy); per-tensor cross-device placement is killed by `-ot` itself (`has_tensor_overrides()` → `pipeline_parallel` → `n_copies == 1`); the first "CUDA + Vulkan cannot serve one model" conclusion was a llama-bench parsing artifact (`-dev A,B` is a SWEEP, not a list — correct `-dev A/B -ts x/y` gives 27.55 / 27.23 tok/s, +7 % decode / +16 % prefill vs iGPU-only).
- Incident — The rootless podman pause process died (trigger NOT established; no OOM kill in the kernel log): every `podman` command failed, all 28 quadlets looped, `podman system migrate` panics in 5.7.0; a reboot repaired it. The reboot exposed `hermes.franksdank.cloud` 502: the Caddyfile dialed `192.168.0.143:8787` (a host bind, not a published port) — fixed to `host.containers.internal:8787`, and the seam that had accepted any numeric upstream was strengthened.
- Host — Model cache cleared on operator instruction (298 → 895 GB free) and re-fetched: halogen's sidecars came back LARGER (in-place versioning — a stale copy is invisible from the file name); Q4_K_S vs Q4_K_M measured +2–3 % with ~5 GiB more host RAM; IQ4_XS fetched as the next comparison payload; `halogen-flash-server.md` gained a "Refreshing the weights" note.
@@ -951,8 +971,8 @@
- `/scripts/frank-medium.env`, `/scripts/serve-frank-medium.sh`, `/scripts/frank-medium.service` — frank-medium: the 3090 left Dankserver, two R9700s (gfx1201) in; runtime charlie12345/ROCmFPX c49ebdb with its self-contained ROCm 7.14 toolchain (bypassing the host's broken mixed ROCm); kingjones Q8_0 AGENT + Q4_0 MTP head; layer split chosen by measurement (1991/18.8 t/s vs tensor 271.8/26.3 — tensor's decode win costs a 7× prefill collapse with no RCCL); draft pinned to ROCm1; frank-small retired (`retire_root_llm_small()` archived the CUDA-era unit).
- `/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md` — New runbook; `freetoken-frank-small.md` and `kingjones-q8agent-rocmfpx.md` marked retired/superseded; AGENTS.md, indexes, System Specs, deployment/Hermes/OpenWebUI/Plex/Stack pages updated.
- Host — Purged 167 stale AMD-repo ROCm 6.3.3 packages, removed the radeon apt pin and the `ld.so.conf` hijack (staged in `~/.AGENTTRASH/20260915-rocm633-cleanup/`).
- `/scripts/frank-large.service`, `/scripts/serve-frank-large.sh` — frank-large cut to ik_llama.cpp (halogen quadlet retired to `~/.AGENTTRASH/20260915-ik-swap/`); overnight decode campaign: 3.8–4.3 t/s across every safe configuration — the wall is ~100 CPU↔GPU section launches per token, not compute; mainline 37b53fd runs 1.0 t/s; the GGUF has no MTP head; 84 GiB of experts cannot fit 24 GiB VRAM (0c1e10a, 795b0b3).
- `/serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md` — New runbook with the full evidence chain.
- `/scripts/frank-dragoon.service`, `/scripts/serve-frank-dragoon.sh` — frank-dragoon cut to ik_llama.cpp (halogen quadlet retired to `~/.AGENTTRASH/20260915-ik-swap/`); overnight decode campaign: 3.8–4.3 t/s across every safe configuration — the wall is ~100 CPU↔GPU section launches per token, not compute; mainline 37b53fd runs 1.0 t/s; the GGUF has no MTP head; 84 GiB of experts cannot fit 24 GiB VRAM (0c1e10a, 795b0b3).
- `/serverAppConfigs-docs/LLM/frank-dragoon-ik-uncensored.md` — New runbook with the full evidence chain.
## 2026-09-14
- `/dankclaw/compose.yaml`, `/dankclaw/.env.example` — New `hermes-webui` service (Hermex): `bootstrap.py --foreground` in the agent image, shares `/home/dank/hermes-data`, loopback-only publish 127.0.0.1:8787, password-gated via `HERMES_WEBUI_PASSWORD`.
@@ -981,7 +1001,7 @@
- `/serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md` — Rewritten to the active-tunnel reality (VM placement, WG gate, socat LaunchAgents, PBKDF2 seed); RDAP proved the "custom WG server" is a ProtonVPN Seattle server (full 65535-port scan: no admin surface anywhere).
- `/dankclaw/authentik/quadlet/authentik-outpost-ldap.container` — New LDAP outpost for Jellyfin (amd64 single-platform digest-pinned — podman 5.7.0 refuses the tag's nested manifest list); authentik objects provisioned via `ak shell` ORM; live bind tests proved two rules: the token intent must be `verification`, and the provider needs the `search_full_directory` object permission.
- `/serverAppConfigs-docs/Infrastructure/Authentik.md`, `/serverAppConfigs-docs/Dankclaw/Jellyfin.md` — LDAP-outpost section (objects table + rules) and the LDAP-Auth 23.0.0.0 plugin section (`LdapSearchAttributes` is comma-separated; username attribute `cn`).
- `/scripts/frank-secondary.env`, `/scripts/serve-frank-secondary.sh` — Two compressed-tensors FP8 swaps attempted on FreeToken and both failed to load (ornith-1.5: expert naming; HauhauCS aggressive: block scales do not broadcast in `_dequant_fp8_weight`); reverted to Huihui FP8 within the hour; `freetoken-frank-small.md` quant matrix updated (modelopt block-fp8 is the only FP8 path that loads); the deploy health gate passed for BOTH failed loads — a swap is only proven by the journal + a real completion.
- `/scripts/frank-zealot.env`, `/scripts/serve-frank-zealot.sh` — Two compressed-tensors FP8 swaps attempted on FreeToken and both failed to load (ornith-1.5: expert naming; HauhauCS aggressive: block scales do not broadcast in `_dequant_fp8_weight`); reverted to Huihui FP8 within the hour; `freetoken-frank-small.md` quant matrix updated (modelopt block-fp8 is the only FP8 path that loads); the deploy health gate passed for BOTH failed loads — a swap is only proven by the journal + a real completion.
- `/dankserver/{romm,jellyfin,calibre-web,copyparty}/quadlet/` — After a hard crash left every bare compose container dead at boot, the media stack standardized to quadlet: six units shipped + cutover handlers (retire bare containers, rescue RomM's anonymous volume, merge-move `/opt/<svc>` → `/home/dank/Containers/<svc>`); v1 generator quirks fixed (`Command=`→`Exec=`, `Device=`→`AddDevice=`, `HealthCmd=` space-splitting, the `HOME=/root` EnvironmentFile trap).
- `/dankclaw/` — Data roots of gitea, authentik, and the hermes trios moved to `/home/dank/Containers/<svc>/` while running (rename is inode-bound — zero downtime); `deploy_hermes` ordering fixed (generated units reload AFTER the unit files are copied).
- `/dankclaw/seerr/quadlet/seerr.container` — New Seerr v3.4.1 request manager: loopback-only 127.0.0.1:5055, guarded vhost + outpost provider `seerr-proxy`; `/tests/seerr_quadlet.bats` new seam; `/serverAppConfigs-docs/Dankclaw/Seerr.md` new page.
@@ -992,8 +1012,8 @@
## 2026-08-30
- `/scripts/frank-npu.env`, `/scripts/serve-frank-npu.sh`, `/scripts/frank-npu.service` — New `frank-npu` on Dankclaw: FLM v1.0.1 serving the whole local registry (36 tags) on 52626; first deploy crash-looped (`/dev/accel/accel0` err=-19) — root cause: the XDNA2 SVA bind fails whenever the AMD IOMMU is off; fixed with `iommu=pt` + reboot; vision + audio towers proven end-to-end; health probe is `/v1/version` (FLM has no `/health` route) (c3ed3a6, cf0c541).
- `/serverAppConfigs-docs/LLM/frank-npu.md` — New runbook; AGENTS.md fleet table + the IOMMU/SVA hard requirement.
- `/dankclaw/frank-large/quadlet/frank-large.container` — frank-large: swapped to `kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF` on the pinned `kingjones30/ROCmFPX` fork (fa-on works UNPATCHED — the QSA F32 assert was agentionai-specific); ~225 s load, real 111k-token prefill at 213–232 t/s.
- `/scripts/frank-secondary.env` — frank-secondary (FreeToken): model swapped to `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` (native block-fp8); ~40 tok/s verified.
- `/dankclaw/frank-dragoon/quadlet/frank-dragoon.container` — frank-dragoon: swapped to `kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF` on the pinned `kingjones30/ROCmFPX` fork (fa-on works UNPATCHED — the QSA F32 assert was agentionai-specific); ~225 s load, real 111k-token prefill at 213–232 t/s.
- `/scripts/frank-zealot.env` — frank-zealot (FreeToken): model swapped to `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` (native block-fp8); ~40 tok/s verified.
- `/serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md` — Reactivated as the large runbook; `agentionai-fast-imatrix.md` marked RETIRED; `freetoken-frank-small.md` model sections updated; LLM index + AGENTS fleet tables refreshed.
- `/serverAppConfigs-docs/Dankserver/prowlarr.md`, `/serverAppConfigs-docs/Dankserver/qbittorrent.md` — Rewritten: the arr↔Prowlarr feed contract (`/{indexerId}/api`, the global key — the hand-rolled `/torznab/` paths were Forms-auth catch-alls returning 302), TV + music chains proven live, first real grab (The Dark Knight 2160p, imported), Prowlarr auto-sync broken by version skew, qB persistent credentials (`WebUI\Password_PBKDF2`, host-local 0600), Readarr parked (abandoned upstream; its client cannot authenticate against qB 5).
@@ -1013,7 +1033,7 @@
- `/tests/caddy_quadlet.bats` (new) + caddyfile/deployment_flow/pihole seams — quadlet boot persistence, self-healing, network, env, and sysctl coverage.
- `/serverAppConfigs-docs/Infrastructure/Caddy.md`, `/serverAppConfigs-docs/Infrastructure/Pi-hole.md`, `/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md`, `/serverAppConfigs-docs/Dankclaw/Gitea.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — quadlet runtime facts, the 2026-08-26 incident, the sysctl-conflict failure mode, canonical Caddy ownership.
## 2026-10-01 — frank-primary RESTORED to fp8 3 x 262144 (bf16 boot fails)
## 2026-10-01 — frank-archon RESTORED to fp8 3 x 262144 (bf16 boot fails)
bf16 (--kv-cache-dtype auto) at 2 x 262144 did not stay up, with OR without
--expert-precise-memory:
@@ -1040,7 +1060,7 @@ KV 6.1e9 -> 792,069 tokens / 3.02x / arena 6,248 / 3.59 GiB free): KV dtype
Restored: fp8 3 x 262144, MNBT 2048, KV_BYTES 6,100,000,000, no precise flag.
Next: bisect bf16 vs SEQS in a window (bf16 + SEQS=3 isolates the dtype).
## 2026-10-01 — frank-primary: --expert-precise-memory + bf16 = boot crash loop
## 2026-10-01 — frank-archon: --expert-precise-memory + bf16 = boot crash loop
**Reverted.** The two changes Frank asked for were applied together and the
result crash-looped: `NRestarts` climbed 1 -> 2 -> 3, each attempt running
@@ -1072,7 +1092,7 @@ overhead: arena 14.31 GiB, ~2.4 GiB free at boot.
default — it is the budget for the phase that allocates last. Reclaim it only
by paying for it elsewhere (smaller pool, fewer sessions, or a shorter context).
## 2026-10-01 — frank-primary: 2 bf16 windows + --expert-precise-memory
## 2026-10-01 — frank-archon: 2 bf16 windows + --expert-precise-memory
Frank's final shape for the night. Two changes at once, both measured in
principle before the boot:
@@ -1097,12 +1117,12 @@ planner's OWN one-max-model-len reserve (2.48 GiB at TP2) and does NOT scale
with SEQS. The pinned `--kv-cache-memory-bytes` therefore does not shrink the
planned arena; it comes out of free VRAM instead.
## 2026-10-01 — frank-large: Q4_K_XL payload + 4 x 163840 sessions
## 2026-10-01 — frank-dragoon: Q4_K_XL payload + 4 x 163840 sessions
Frank's call. Payload swapped from ajgazin's **abliterated** Swift-1.5
UD-Q8_K_XL (29.3 GiB) to **unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL** (16.35 GiB,
sha256 3f227079…b01e, vanilla/Apache-2.0 — the endpoint is no longer
uncensored). New fetcher `scripts/download-frank-large-q4.sh`; the DFlash2
uncensored). New fetcher `scripts/download-frank-dragoon-q4.sh`; the DFlash2
drafter and its fetcher are unchanged.
Shape: **`--sessions 4 --context 163840`** with unquantized KV. gufo has no
@@ -1136,12 +1156,12 @@ deployer's `DEPLOY INCOMPLETE` banner can be raised by a flake. Re-run
stable underlying issue and it lives in Plex's web-client/image config, not in
any repo service.
## 2026-10-01 — frank-primary bf16 no-spec cutover
## 2026-10-01 — frank-archon bf16 no-spec cutover
- `/scripts/frank-primary.env` — bf16 no-spec cutover, FINAL shape after the
- `/scripts/frank-archon.env` — bf16 no-spec cutover, FINAL shape after the
night ladder: MAXLEN **241664** x SEQS 3, KV_BYTES **10,184,000,000**/rank,
KV_DTYPE auto (bf16), MNBT 1024, GATHER CHUNKS=32/LANES=64, UTIL 0.98;
FRANK_PRIMARY_SPEC deleted (no spec). Engine measured at the winner:
FRANK_ARCHON_SPEC deleted (no spec). Engine measured at the winner:
"KV cache size: 732,359 tokens, Maximum concurrency for 241,664 tokens per
request: 3.03x".
- **The night ladder (Frank's method).** 262144 OOMed at CUDA graph capture
@@ -1156,10 +1176,10 @@ any repo service.
- **Util stays 0.98.** The earlier 0.90 edit was wrong — lowering
`--gpu-memory-utilization` shrinks the engine's own budget rather than
buying slack; Frank forbade it and the winning ladder ran at 0.98.
- `/scripts/serve-frank-primary.sh` — `--speculative-config` line removed;
- `/scripts/serve-frank-archon.sh` — `--speculative-config` line removed;
`--max-num-batched-tokens` and the `-e R4D_LRU_GATHER_CHUNKS/LANES`
container-env passthrough added; shape header rewritten.
- `/tests/frank_primary_unit.bats` — seam re-pinned: bf16-KV-present,
- `/tests/frank_archon_unit.bats` — seam re-pinned: bf16-KV-present,
spec-absent-by-measurement, gather-32x64-present, MNBT-1024 tests; the
qualified-shape and flags tests flipped to the new env keys.
- Overnight redo campaign (prefetch OFF everywhere) tabled in
@@ -1168,7 +1188,7 @@ any repo service.
1024-4096, KV pool law (7,926 fp8 / 14,047 bf16 B/token), arena flat zone,
six-slot refusal, bf16 3x262144 boot failure root cause (CUDA OOM at
KV_BYTES 11.3e9; boots clean at 11.047e9 — captured from journald `-o json`).
- `/AGENTS.md` — fleet table frank-primary row rewritten to the bf16 shape.
- `/AGENTS.md` — fleet table frank-archon row rewritten to the bf16 shape.
- `/serverAppConfigs-docs/LLM/index.md` — fleet intro + mermaid node
updated out of the GLM era.
- **Client contract fixed (thinking + tools).** The live endpoint returned a
@@ -1204,7 +1224,7 @@ any repo service.
knife-edge shape change, send a large-`max_tokens` request before calling it
done. FOUR slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93
GiB per rank for activations/graph capture. Frank's call 2026-10-01.
- `/scripts/frank-primary.service` — **Restart=always, not on-failure** (plus
- `/scripts/frank-archon.service` — **Restart=always, not on-failure** (plus
`StartLimitIntervalSec=0` and `OOMScoreAdjust=-500`). A runtime worker fault
makes vLLM's API server exit CLEAN, so ExecStart returned 0, systemd logged
"Deactivated successfully", and on-failure NEVER fired: the endpoint stayed
@@ -1224,20 +1244,20 @@ any repo service.
- Production bring-up: the shape boots and serves on :7770 — engine log
verified (KV cache size, concurrency, MNBT 1024, gather 32x64) plus live
contract probes (reasoning split, tool_calls). Rollback target for the
2026-09-29 fp8+MTP4 shape: git tag `frank-primary-fp8-mtp-20261001`.
2026-09-29 fp8+MTP4 shape: git tag `frank-archon-fp8-mtp-20261001`.
## 2026-10-01 — frank-large cutover: gufo -> vLLM TheRock, MiMo-V2.6 BF16 + cross-model MTP
## 2026-10-01 — frank-dragoon cutover: gufo -> vLLM TheRock, MiMo-V2.6 BF16 + cross-model MTP
- **Question that started it:** can a Qwen3.5 MTP drafter attach to
frank-secondary, and how does vLLM run the FULL BF16 MiMo checkpoint? Answer 1:
not on frank-secondary — SGLang reads the MTP head from the TARGET checkpoint
frank-zealot, and how does vLLM run the FULL BF16 MiMo checkpoint? Answer 1:
not on frank-zealot — SGLang reads the MTP head from the TARGET checkpoint
and MiMo ships ZERO `mtp.*` tensors (base and INT8 alike). Answer 2: the
digest-pinned `kyuz0/vllm-therock-gfx1151` image (vLLM 0.30 / torch 2.11 /
TheRock ROCm 10) boots BF16 MiMo on the gfx1151 iGPU in ~200-240 s, and
vLLM reads the draft from a SEPARATE path — so cross-model MTP from
`Qwen/Qwen3.5-9B` (the distill's source, identical 4096/248320/32-layer
geometry) works. Frank's call: ship that as frank-large.
- **Bench window (frank-large stopped, one session, temp 0, 28,647-token
geometry) works. Frank's call: ship that as frank-dragoon.
- **Bench window (frank-dragoon stopped, one session, temp 0, 28,647-token
prefill / 512 decode):** base 17,876 t/s prefill / 13.1 t/s decode (13.1
is ~94% of the LPDDR5x BF16 roof - decode here is bandwidth, not a bug);
+DFlash-8 7,976 / 17.4 (acceptance 59%/26% per position); +MTP-2 7,380 /
@@ -1248,7 +1268,7 @@ any repo service.
cascade). vLLM spec decode is distribution-preserving, not
bitwise-greedy-preserving.
- **Client-contract delta wired:** vLLM emits `reasoning`, not
`reasoning_content` - `models.yml` frank-large block now sets
`reasoning_content` - `models.yml` frank-dragoon block now sets
`reasoningContentField: "reasoning"`, and the extraBody moved to the
Qwen3.5-family recipe (0.7/0.8) with `chat_template_kwargs`
(a top-level `preserve_thinking` is ignored by vLLM).
@@ -1260,7 +1280,7 @@ any repo service.
0.30 (verified in-image); the drafter stays on disk as the alternative.
- Host state: +38 GiB fetched (MiMo BF16 17.53 GiB, Qwen3.5-9B 18 GiB,
DFlash 2.58 GiB), disk down to ~26 GiB free - reclaim pass due. gufo
rollback pinned at git tag `frank-large-gufo-q4kxl-20261001`; engine,
rollback pinned at git tag `frank-dragoon-gufo-q4kxl-20261001`; engine,
Q4_K_XL payload and DFlash2 drafter all remain on the host.
- First deploy attempt #2 crash-looped the container 51 times: the serve
script handed vLLM the HOST model path while the tree is mounted at
@@ -1269,14 +1289,14 @@ any repo service.
already converted; the target was not), pinned by bats. Also fixed the
deployer's own draft gate (same single-file bug as the serve script's).
- Reclaim (Frank approved): Swift-1.5 Q8_K_XL payload (30 G, refetchable via
`scripts/download-frank-large-swift.sh`) and the orphaned flux-2-klein
`scripts/download-frank-dragoon-swift.sh`) and the orphaned flux-2-klein
tree under `models/checkpoints` (23 G; ComfyUI mounts
`comfyui-data/models`, not this) purged via
`~/.AGENTTRASH/20261001-frank-large-reclaim/` — 52 G freed. KEPT the
`~/.AGENTTRASH/20261001-frank-dragoon-reclaim/` — 52 G freed. KEPT the
1.1 G `Qwen3.8-27B-DFlash2-Q4_K_M.gguf` (it is the gufo rollback's
drafter, not a remnant).
## 2026-10-02 — frank-large cutover: vLLM MiMo -> ROCmFPX Q8_0_ROCMFPX_AGENT + MTP
## 2026-10-02 — frank-dragoon cutover: vLLM MiMo -> ROCmFPX Q8_0_ROCMFPX_AGENT + MTP
Frank's order was the highest quality of
`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` on one full context, "whatever
@@ -1296,16 +1316,16 @@ bundled BF16 mmproj, and MTP from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf`
output-head tensors at plain Q8_0 instead of 2, which measured MTP acceptance
0.953 vs 0.911.
- Steady GTT occupancy 63.1 GiB of the ~114 GiB aperture; boot to healthy 16 s.
- Rollback: git tag `frank-large-gufo-q4kxl-20261001`. The retired vLLM-era
- Rollback: git tag `frank-dragoon-gufo-q4kxl-20261001`. The retired vLLM-era
fetchers moved to `old/`. Runbook
[[frank-large-rocmfpx-q8agent-mtp]].
[[frank-dragoon-rocmfpx-q8agent-mtp]].
## 2026-10-02 — frank-secondary: Q5_K_M, then three contexts + vision, sized by two boots
## 2026-10-02 — frank-zealot: Q5_K_M, then three contexts + vision, sized by two boots
Three orders in one day on the same endpoint: `bartowski/…Q5_K_M` on llama.cpp
with two contexts ("whatever context size fits the 3090"), then **three
contexts** "as much as can fit while keeping overhead open space to 2-3gb on
the 3090", then **"frank-secondary should have vision"**.
the 3090", then **"frank-zealot should have vision"**.
- **Final shape: `CTX_SIZE=540672 PARALLEL=3` = 3 x 180,224**, f16 K + q8_0 V
under flash attention, `--mmproj` vision, all layers on the 3090. Measured
@@ -1321,9 +1341,9 @@ the 3090", then **"frank-secondary should have vision"**.
- **Three contexts cost window, and that is the tradeoff Frank was told:**
180,224 tokens per conversation, not the native 262,144. A longer prompt is
a hard `400 exceed_context_size_error`, so `models.yml` sets
`contextWindow: 180224`. Consequence at the time: frank-primary's then
`contextWindow: 180224`. Consequence at the time: frank-archon's then
183,501-token compaction request did not fit here with summary room —
resolved later the same day when frank-primary's window moved to 200,704
resolved later the same day when frank-archon's window moved to 200,704
and its trigger fell to 140,492 (see the shape-D entry below).
- Measured (2026-10-02, this shape): decode **106.8 t/s**, prefill **3,814
t/s** (~5 % faster decode than 2 x 262,144 — shorter slot, shorter attention
@@ -1343,11 +1363,11 @@ the 3090", then **"frank-secondary should have vision"**.
`~/.AGENTTRASH/20261002-retired-llm-payloads/` (~59 GiB of superseded
payloads). Kept in place for rollback: `MiMo-Q6K/`, `MiMo-Q5KM/`,
`Qwen3.8-27B-UD-Q4_K_XL/` and `Qwen3.8-27B-UD-Q8_K_XL/` (the latter holds the
gufo DFlash2 drafter). Runbook [[frank-secondary-llama-mimo-q5km]].
gufo DFlash2 drafter). Runbook [[frank-zealot-llama-mimo-q5km]].
## 2026-10-02 — frank-primary shape D: 3 x 200704, fp8, pool sized for 2 contexts
## 2026-10-02 — frank-archon shape D: 3 x 200704, fp8, pool sized for 2 contexts
Frank's order: shrink frank-primary to a 200704-token window with about two
Frank's order: shrink frank-archon to a 200704-token window with about two
resident full-length contexts, cut the KV pool, and drop MNBT to 1024. The
first attempt (`30904ba`, 2 x 200704 + SEQS=2) **crash-looped**; the revert
went live as `e460bbb`, and the shape was then re-applied one law at a time on
@@ -1412,13 +1432,13 @@ top of the proven-good base, root cause first.
400 for anything else (re-measured; the fork's message names the set).
- **Harness contract followed the window:** `models.yml` `contextWindow`
262144 -> **200704** (compaction trigger 140,492 + 36,864 = 177,356 <=
200,704). Side effect: frank-primary's compaction request now fits inside a
frank-secondary slot again, which it had stopped doing when frank-secondary went to
200,704). Side effect: frank-archon's compaction request now fits inside a
frank-zealot slot again, which it had stopped doing when frank-zealot went to
3 x 180,224.
- **Comment-only follow-up** `295d578` recorded the measured numbers and was
deployed so the per-service revision marker sits at HEAD; e2e 33/33 again.
- Rollback for the pre-shape-D fp8+MTP4 era stays git tag
`frank-primary-fp8-mtp-20261001`. Runbook
`frank-archon-fp8-mtp-20261001`. Runbook
[[dankserver-qwen38-fp8-expert-offload-tuning]].
## 2026-10-02 - plex: the readiness gate probed HTTP against a TLS-only service
@@ -1452,12 +1472,12 @@ and then answers with nothing, and curl reports "Empty reply from server" (52).
- Re-deployed `39f0e0d`: plex recorded deployed, the *arr services recorded
(compose-recorded on this host, no restart attempted), tautulli deployed,
e2e 33/33. Verified after: `plex-https=200`, plex container up, tautulli
healthy, frank-primary untouched (`health=200`, NRestarts=0).
healthy, frank-archon untouched (`health=200`, NRestarts=0).
- **Operating lesson: `deploy.sh <target>` with no `--service` is a fleet-wide
drain.** It restarts every service whose recorded revision is behind HEAD and
whose paths changed since. To touch one service, name it.
## 2026-10-02 — frank-primary shape E: `--expert-precise-memory` (+1,026 arena slots), a mounted compile cache, and the P2P answer
## 2026-10-02 — frank-archon shape E: `--expert-precise-memory` (+1,026 arena slots), a mounted compile cache, and the P2P answer
Shape D left one question open — the plan's overhead term was reserving 2.35 GiB
of compile headroom the boot never used — and the answer to Frank's free-VRAM
@@ -1466,7 +1486,7 @@ every restart since the cutover.
- **`--expert-precise-memory` is real and it fits (option A).** Boot-probed on
the shape-D base with production stopped, the candidate booted AS the service
(`FRANK_PRIMARY_ENV_FILE=/tmp/probe-b.env bash scripts/serve-frank-primary.sh`,
(`FRANK_ARCHON_ENV_FILE=/tmp/probe-b.env bash scripts/serve-frank-archon.sh`,
trap-guaranteed restore), the numbers read off the engine's own plan lines:
| plan line | shape D | shape E |
@@ -1495,7 +1515,7 @@ every restart since the cutover.
range (1, 1024) takes 173.19 s` ... `torch.compile took 181.20 s in total` ...
`init engine (profile, create kv cache, warmup model) took 217.26 s
(compilation: 181.20 s)`. The serve script now mounts
`FRANK_PRIMARY_CACHE_DIR` at `/root/.cache` — the **parent**, so the vLLM
`FRANK_ARCHON_CACHE_DIR` at `/root/.cache` — the **parent**, so the vLLM
AOT/torch_compile tree (`VLLM_CACHE_ROOT`, default `/root/.cache/vllm`) and
the comgr JIT cache both persist, and nothing is hidden because the image has
no such directory. Same class of fix as the recipe's