Frank
9cf4b1de82
PORTMAP=7773:7770 hook; health 200 on 127.0.0.1 and 192.168.0.235, decode through the published path in band. 7773 chosen over 7770 (frank-primary owns 7770 when it returns). Log bullet recovered from a misplaced insert into the 2026-09-28 cleanup section.
519 lines
60 KiB
Markdown
519 lines
60 KiB
Markdown
# Bundle Change Log
|
||
|
||
|
||
## 2026-09-29
|
||
- **frank-large tuning investigation: a measured NEGATIVE result, nothing
|
||
changed.** Frank asked for iterative decode/prefill improvement at a 32k
|
||
working point without touching the model or quantization. **No win exists
|
||
at gufo's served settings**, so no config change was shipped. frank-large
|
||
was stopped for the whole investigation (~4h) per the benchmarking policy
|
||
and restored healthy in 9s afterwards.
|
||
- Measured on the repo's own prose and shell sources at **32,762 prompt
|
||
tokens**, salted per run so a warm prefix cache could not flatter a
|
||
later run, 3–5 repetitions per configuration.
|
||
- **`--prefill-chunk` is dead**: 512 → 8192 moves prefill 410.7–412.4 t/s.
|
||
A 16x change in chunk size, 0.4% out.
|
||
- **`--draft-tokens` is flat**: 5 → 10 gives 13.79–15.86 t/s decode, all
|
||
inside the run-to-run noise band. The 5-rep confirmation round
|
||
**reversed** the 3-rep leaders — `dt9 + chunk4096` looked best and came
|
||
**last** (12.86), while the **deployed config came first** (14.71).
|
||
- `--draft-policy` fixed vs adaptive: no effect. `--min-draft-tokens`:
|
||
**rejected by the engine** ("DFlash2 requires ... 1"). MTP: **unsupported
|
||
for Q8 targets** (`Unsupported Qwen MTP tensor type for
|
||
token_embd.weight: Q8_0`).
|
||
- `--context 262144 → 40960` was tested as a diagnostic (the deployed value
|
||
preallocates KV ~8x larger than a 32k conversation needs): **no gain**
|
||
(13.93 vs 14.71 t/s), and it would break `models.yml`'s declared
|
||
`contextWindow: 262144`. Not shipped.
|
||
- **What actually moves the number is speculation itself: 6.65 t/s off →
|
||
~15 t/s on, 5.3x** — already enabled. Prefill is insensitive to
|
||
everything tried, consistent with gufo's own `EXPERIMENTS.md` where
|
||
dozens of kernel variants are recorded rejected for "no release
|
||
throughput improvement".
|
||
- **Two traps are now in AGENTS.md's benchmarking policy**, because they
|
||
cost most of the wall time and both produced *false* results:
|
||
(1) **`gufo bench` cannot rank speculative settings and inverted the
|
||
ranking** — its corpus reports `acceptance=0.933 +/- 0.000` with an
|
||
identical trace sha256 across draft widths (the same highly-predictable
|
||
text every run, 93% of proposals accepted). It reported 29–37 t/s where
|
||
real traffic gives ~15, and made dt16 look 6% faster than dt7 when real
|
||
text made it 14% slower. (2) **A live A/B needs >=128 generated tokens**;
|
||
at 8, `inter_token` is first-token noise and the first pass of the sweep
|
||
produced a nonsense ordering that the 128-token pass reversed.
|
||
- Also worth recording: the `Server-Timing` header is
|
||
`ttft;dur=402.485` — the unit is in the **name** half, so splitting on
|
||
`=` keys the dict `ttft;dur` and every lookup silently returns nothing.
|
||
Four configurations reported "no Server-Timing" while the header was
|
||
present on all of them.
|
||
- **dankserver FP8 decode-tuning campaign: flags exhausted at 26.8 tok/s,
|
||
mechanism measured.** The ad hoc qwen-fp8 container (tcclaviger RDNA4
|
||
fork, orcarouter Qwen3.8-Flash-Next-Uncensored-FP8, TP=2) ran a full
|
||
flag-level A/B battery at the 32k working point; every number is from the
|
||
server's usage counters (SSE chunks are invalid under spec decode). New
|
||
runbook: [[dankserver-qwen38-fp8-expert-offload-tuning]].
|
||
- Incumbent stands: MTP depth 4, stock gather 8x16, prefetch depth 1,
|
||
TUNABLE off — **26.80/25.15/26.81 tok/s** at 166/1087/4064 ctx,
|
||
acceptance 60.7%; the re-check reproduced it (25.80/24.38/26.63).
|
||
- **Ruled out by measurement:** gather geometry 16x32 (−9%) and 32x64
|
||
(parity) — the fork's "tuned at the Gen4 ceiling" comment is stale, the
|
||
stock constants are already right on Gen5 x8; MTP depth 3 (−9%, flat in
|
||
ctx) and depth 6 (collapses to 13.5–19, acceptance 17.4% — and proves
|
||
the UVA read-through path is SLOWER than copy-in); MNBT 8192 (neutral);
|
||
prefetch depth 2 (neutral — the tier is 16 slots/layer against a
|
||
~2,400-slab working set at 73% misses); 2-way concurrency (summed
|
||
25.49 < single 26.8 — under TP both streams' misses gather on each
|
||
rank, the links do not add capacity, and the shared arena thrashes).
|
||
- **TunableOp rejected:** it silently skips under expert offload unless
|
||
`CLAV_TUNABLEOP_WITH_OFFLOAD=1`; once forced on it corrupts numerics —
|
||
repetition loops, the server's repetition-halt firing at ~140 of 256
|
||
tokens, "79.8 tok/s" measured on halted degenerate completions, and a
|
||
fresh-prompt check at 20.01 (below baseline). +6.7 min warmup, no
|
||
persistent cache in the container lifecycle.
|
||
- **Infeasible:** `UTIL=0.99` (free-memory wall: 31.29 of 31.86 GiB free
|
||
at worker init); `--max-num-seqs 2` (compile constraint violation at
|
||
CUDA-graph capture); ngram-only drafter (fork bug: "PLE inputs were not
|
||
prepared" — PLE prep is wired only for the MTP spec path).
|
||
- **The wall, measured:** the link does Gen5 x8 = 31.5 GB/s/direction;
|
||
the HIP probe reads 22.74 GB/s at the exact 2.345 MiB expert slab (91%
|
||
of ceiling at bulk); the engine's effective gather rate is ~9.5 GB/s =
|
||
42% of that. A decode step is ~100 ms compute + ~400 ms PCIe wait
|
||
(peak util 100%, smooth ~29%). The gap to the 35 tok/s target is the
|
||
fork's gather read path — fork-patch territory, not flags.
|
||
- **Telemetry fork bugs:** the torch-profiler worker stop hangs on ROCm
|
||
(>10 min, two attempts, while the AsyncLLM wrapper logs success in
|
||
1 s), and the default `torch_profiler_with_stack=true` grew a 6.5-min
|
||
profile window's trace buffer ~40 GB → the kernel OOM-killed the engine
|
||
core (anon-rss 95.2 GB, `oom-kill` in `journalctl -k`). The per-layer
|
||
expert-cache hit rates (`R4D_LRU_TELEMETRY=1`) are reachable ONLY
|
||
through that stop path, so the telemetry JSON is unreachable without a
|
||
fork patch.
|
||
- **256k ctx boots** (the last original-goal dimension): a 262,144-token
|
||
window needs **4.0 GiB of KV** (hybrid attention + mamba pages ≈
|
||
16 KiB/token, not the 32k plan's 9,190 B/token) — 3 GiB boot-refused
|
||
(`4.0 GiB needed > 2.98 available`; the engine's own max-len estimate
|
||
at that pool: 184,224). At KV 4.5 GiB it serves **36.65/34.21/34.77**
|
||
(arena 6,574→5,648 slots, −14%, ~10% decode cost; 2 of 3 depths just
|
||
under 35). KV-exact-4.0 arm in flight for the last +3.5% slots.
|
||
Correction recorded: the planner sizes the arena AFTER the KV override
|
||
(the earlier "before" reading was wrong) — the arena is what remains.
|
||
- **Final 256k probe: the KV trim is inert.** KV-exact 4.0 GiB
|
||
boot-refused by 20 MB of accounting (3.98 available < 4.0 needed;
|
||
engine estimate at 3.98: max len 260,176); +32 MiB (`KV_BYTES=
|
||
4328521728`) boots — but the arena is **identical to the 4.5 GiB arm
|
||
(5,648 slots)**: the planner buckets, the 0.47 GiB went to margin.
|
||
Decode band **36.48/34.13/35.02** (acceptance 39.4%) — same 34.1-36.5
|
||
band, straddling the 35 target within noise. Final 256k config:
|
||
`MAXLEN=262144 KV_BYTES=4328521728`, prefetch off. The full-speed
|
||
alternative stays `MAXLEN=184224` at 3 GiB (37.4-40.6 band).
|
||
- **GLM-5.3-Flash readiness REVISED by source reading (Frank's question
|
||
about int4 vs mxfp4):** the tcclaviger fork in the qwen-fp8 container
|
||
registers `Glm5NextForCausalLM`/`Glm5NextMTP` with a backend-split
|
||
module (`glm5next/amd|common|nvidia`, `sparse_indexer.py` for DSA);
|
||
`offloader/expert.py` accepts `R4dDsfp4MoEExperts, R4dW4a16MoEExperts,
|
||
Fp8HipMoEExperts` — and **`R4dDsfp4MoEExperts` is MXFP4 weights with
|
||
the DeepSeek activation law** (docstring; SwiGLU `gemm1_clamp_limit`
|
||
default 10.0), inheriting the mxfp4 repack/GEMM/emulation-fallback.
|
||
**`r4d_lib.has_dsfp4()` is True** in this build — native `moe_dsfp4` /
|
||
`moe_dsfp4_mt1` entries exist, so the old "MXFP4 = emulation, missing
|
||
kernels" GLM verdict is dead for this fork. WNA16 (int4) backend
|
||
oracle on ROCm reduces to HUMMING/TRITON (Marlin/TRTLLM/XPU are
|
||
CUDA/NVIDIA/Intel-only), payload must be compressed-tensors W4A16
|
||
group-wise symmetric. Remaining GLM gate: a compressed-tensors MXFP4
|
||
(~165 GiB) or W4A16 payload + a glm5next-on-HIP boot probe; community
|
||
GLM-5.3 quants are GGUF (llama.cpp path only), native FP8 is 328 GB
|
||
(> 246 GiB RAM). Feasibility doc rows updated.
|
||
- **Revert pins installed (Frank's request):** (1) repo tag
|
||
`qwen-fp8-tuned-20260929` at f532e03 (the campaign's docs + measured
|
||
config; `git checkout qwen-fp8-tuned-20260929` restores the state);
|
||
(2) image pinned by digest — serve script now launches
|
||
`docker.io/tcclaviger/vllm@sha256:da402966...` (ID 3322dbbbc96d) with a
|
||
local alias `qwen-fp8-runtime:tuned-20260929`; the floating `:latest`
|
||
is no longer referenced anywhere (the other image on the host,
|
||
`28.06.1-Code000` = fffca39f, is a DIFFERENT build — do not confuse).
|
||
Runbook deployment-shape section updated.
|
||
- **Endpoint published:** `PORTMAP=7773:7770` hook added to the serve
|
||
script; the 256k config relaunched with it. Host-side checks: health
|
||
200 on 127.0.0.1:7773 AND 192.168.0.235:7773, /v1/models lists
|
||
qwen3.8-flash-next, decode through the published path
|
||
36.49/34.55/35.98 (same band). 7773 chosen over 7770 — frank-primary
|
||
owns 7770 when it ever returns.
|
||
|
||
## 2026-09-28
|
||
- **Dankclaw model cleanup** (disk was at 93%, 137 G free). Frank asked to
|
||
clean up old models. **510 G staged** to
|
||
`~/.AGENTTRASH/20260928-retired-models/` — moved, NOT deleted, with a
|
||
`MANIFEST.md` giving a restore command per item. **Nothing is freed until
|
||
the purge is approved**; that is the point of the trash policy, and the
|
||
number to know before expecting the disk to move.
|
||
- Staged: CIRU Orca 128 G, halogen 123 G, unsloth Flash-Next IQ4_XS 91 G,
|
||
mradermacher IQ4_XS 91 G, unsloth 27B 29.3 G, Kairic Edge 27 G,
|
||
ISTA-DASLab GSQ-RCO 13 G, vLLM Gemma4 w4a16 9.6 G, vLLM Gemma4 drafter
|
||
838 M. Every one maps to a runtime already retired in the runbook index.
|
||
- **The 29.3 G rollback target went with them**, by Frank's decision. The
|
||
payload-only rollback is now a re-fetch first; both the runbook and
|
||
AGENTS.md say so. **The DFlash2 drafter was NOT reclaimed** — only the
|
||
`.gguf` was taken out of that directory, because the live frank-large
|
||
drafter lives in it. The manifest warns against restoring the directory
|
||
wholesale.
|
||
- **Kept deliberately:** `checkpoints/` 23 G. It is unreferenced by the
|
||
repo, but ComfyUI is live and its workflows were not searched — not
|
||
confirmed dead, so not taken. `logs/x4e-run.log` also stays: it is a
|
||
**sparse** file, 3.3 G apparent but 4 K real, so removing it frees
|
||
nothing.
|
||
- **A live reference check came first, and it earned its keep.**
|
||
`serve-frank-small.sh`'s *fallback* `MODEL_DIR` still pointed at the
|
||
retired GSQ-RCO payload frank-small served on 2026-09-16/17. The env
|
||
file has always overridden it, so the drift was invisible — but a
|
||
missing or unreadable env file would have turned a config problem into a
|
||
"missing asset" naming a model that no longer exists. Repointed at
|
||
`Gemma4-12B-QAT/`, which is what the env file already sets. Every other
|
||
hit was a comment, a retired fetch script's destination, or
|
||
`retire_halogen_large_quadlet` (which moves a *quadlet*, not a model).
|
||
- **Not taken, by Frank's decision — scope was models only:** podman
|
||
images (**279 G reclaimable, 95%**; 170 images, 23 active) and the
|
||
uv+pip caches (**73 G**). Together that is 352 G of regenerable garbage
|
||
larger than the models reclaimed, and it is the obvious next move.
|
||
- Pre-existing: **118 G of trash from 2026-09-21** was already staged
|
||
(`20260921-frank-large-assembled` 88 G, `-redownload` 30 G) and never
|
||
purged. Total now staged on this host: **627 G**.
|
||
- Verified after the moves: frank-large and frank-small both `active`,
|
||
both `/health` 200, and all three live assets (Swift target, DFlash2
|
||
drafter, Gemma4 target) still present.
|
||
- **frank-large payload swapped to Swift-1.5 (uncensored)** (production
|
||
change, Dankclaw). Frank asked whether the UD Q8 of
|
||
`ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` would load on
|
||
gufo, then said to do it. Port 7770, alias `frank-large`, one 262,144
|
||
session, the DFlash2 drafter and the client contract are unchanged; the
|
||
27B dense weights are an **abliterated (uncensored)** fine-tune of the
|
||
same base, under the **non-OSI SWIFT OPEN LICENSE 1.0**.
|
||
- **Compatibility was established from the GGUF headers, not from the
|
||
filename.** Both files were range-fetched and parsed: same
|
||
`general.architecture` (`qwen35`), same 866 tensors with **identical
|
||
names in identical order**, **zero** per-tensor `ggml_type` or dim
|
||
mismatches (Q8_0×453, F32×360, BF16×53), same MTP head at `blk.64`, same
|
||
geometry. The two scary metadata differences are both no-ops, confirmed
|
||
against gufo's source: `general.file_type` (15 vs 7) is *"deliberately
|
||
not consulted"* by `gguf_reader.cpp` — the file gufo already served
|
||
records 15 (Q4_K_M) while containing no Q4_K tensor — and Swift's extra
|
||
`qwen35.attention.recurrent_layers` key is **never read** by gufo
|
||
anywhere in `src/`; layer kinds come from `full_attention_interval`,
|
||
which both files carry as `4` and which agrees with it.
|
||
- **Fetched and verified:** 31,457,990,272 bytes, sha256 `9076f99a…`
|
||
matching the Hub LFS oid, via the new
|
||
`/scripts/download-frank-large-swift.sh`. It sends **no auth header** —
|
||
the repo is public and ungated, and a token on the curl command line is
|
||
world-readable via `/proc/<pid>/cmdline`. The anonymous transfer ran at
|
||
~1.8 MiB/s; resuming with the host's token through a **mode-600
|
||
`curl -K` config** took it to ~7 MiB/s. The download survived an ssh
|
||
drop by running `setsid nohup`, not under the session.
|
||
- **Load test, service stopped first** (the GPU policy): **9.2 s** from
|
||
`load_started` to `load_completed`, 66.5 GiB of the 116.7 GiB iGPU at
|
||
context capacity 262,144, no loader warnings. Contract battery green:
|
||
`context_length 262144`, `content` + `reasoning_content`,
|
||
`finish_reason: tool_calls`, SSE, and a **two-turn tool loop** with prior
|
||
reasoning preserved. Spot-checks: "17*23" → `391`; asked for a
|
||
lockpicking scene it wrote one without hedging, confirming the
|
||
abliteration.
|
||
- **THE TEMPLATE LAW IS GONE, by decision.** Frank asked to "use our chat
|
||
template"; gufo **cannot** take it. This is not a missing flag —
|
||
`src/models/qwen/chat_template.cpp` is 587 lines of hand-written C++
|
||
that appends strings, with no Jinja engine, no `--chat-template-file`,
|
||
no env override and no template body in the binary. Qwen-Sharp
|
||
v22.4.0 is 453 lines of Jinja2 with `namespace()`, macros and
|
||
`raise_exception`, so supplying it is a **port, not a config line**.
|
||
Frank chose to ship on gufo's stock renderer and accept the loss: the
|
||
`terse` system-prompt injection, in-band `<|think_*|>` control, the
|
||
`json|xml` `tool_call_format` switch, tool-argument truncation and
|
||
`add_vision_id` prefixes are all gone. gufo's renderer does keep the
|
||
effort instruction strings and `preserve_thinking`. **Worth watching:
|
||
an abliterated model now runs without the prompt that tells it to stop
|
||
waffling** — the cheap fix is to send the `terse` text as an ordinary
|
||
system message from `models.yml`, not to touch a template.
|
||
- **The payload is now two files in two directories.** The target moved
|
||
repo; the DFlash2 drafter did not move, so `DRAFTER_DIR` is its own
|
||
tunable. That broke the deployer's stray-run sweep, which keyed on the
|
||
*target* filename and would have silently stopped matching — a stray
|
||
run would then have held 7770 past the port gate. It now keys on the
|
||
drafter, which every frank-large command line carries via
|
||
`--dflash-model` and which did not move.
|
||
- **`/tests/frank_large_unit.bats`** — 20 assertions. The mutation pass
|
||
found **four real gaps** on the first run, all now fixed: two fetch
|
||
assertions were satisfied by a **header comment** rather than the URL
|
||
assignment, so repointing a repo left them green; and nothing bound the
|
||
env's `MODEL_FILE`/`DRAFTER_FILE` to the filenames the fetch scripts
|
||
actually download, so a re-upload or an env edit would break the
|
||
endpoint with a "missing asset" no test predicted. There is now an
|
||
explicit cross-file invariant. A second harness bug worth recording:
|
||
a `bats --filter` that matches no test reports zero `not ok`, which
|
||
looks exactly like a mutation that survived — the harness now asserts
|
||
the `1..N` count before believing a result. 16/16 mutations caught.
|
||
- Docs: runbook gained a **Payload** section (the header evidence),
|
||
**The template law** section and a two-way **Rollback**; AGENTS.md
|
||
fleet table, frank-large section and rollback.
|
||
- **frank-large runtime swapped to gufo** (production change, Dankclaw).
|
||
Frank asked for `github.com/gufo-org/gufo` — a vertical inference engine
|
||
built for exactly this hardware (Ryzen AI MAX+ 395 / Radeon 8060S,
|
||
`gfx1151`) — in place of the stock llama.cpp prebuilt. Port 7770, alias
|
||
`frank-large`, one 262,144-token session and the client contract were all
|
||
unchanged; only the engine was new, and the payload was the unsloth
|
||
Qwen3.8-27B UD-Q8_K_XL **at that point** — it was swapped for the
|
||
abliterated Swift-1.5 later the same day, see the entry above.
|
||
- **The runtime is built from source on the host.** gufo ships no prebuilt
|
||
Linux binary (the v0.1.0/v0.1.1 release assets are empty) and the host
|
||
has no Nix. Pinned at `v0.1.1` / `b0f8673`, installed to
|
||
`/home/dank/GitPackages/gufo-v0.1.1/` by the new
|
||
`/scripts/build-gufo.sh`, which is reproducible and fails closed on a
|
||
moved tag. Unlike the unsloth prebuilt it replaces, gufo **links the host
|
||
ROCm 7.2.4** through its RPATH, so the HIP dev packages must stay.
|
||
- **Two host workarounds, both encoded in the build script.** Ubuntu's
|
||
`librocwmma-dev` ships a truncated rocWMMA header set into
|
||
`/usr/include` that shadows AMD's complete one; the build died on
|
||
`'internal/accessors.hpp' file not found`, and a `-I` cannot fix it
|
||
because in HIP mode the host compilation pass searches the system
|
||
directories first. The package has no reverse dependencies; it is
|
||
removed. Second, Ubuntu 26.04 ships `/usr/include/c++/16` and the ROCm
|
||
clang prefers the newest tree, whose `<atomic_wait.h>` needs a
|
||
`std::__detail::__notify_impl` symbol version the installed
|
||
`libstdc++.so.6` (GLIBCXX_3.4.35) does not export — the link failed until
|
||
`--gcc-install-dir` was pinned to GCC 15.
|
||
- `gufo diagnose` reports `[PASS]` on `gfx1151` (ROCm 7.2.3, amdgpu
|
||
loaded, 20 CUs). It self-caps compute queues at 2 resident for
|
||
`serve llm` after reading `/sys/class/kfd/kfd/proc/*/queues/*/type`,
|
||
which matters because ComfyUI holds queues on the same device.
|
||
- **Shape:** one session, `--context 262144`, DFlash2 with the **adaptive**
|
||
block policy capped at `--draft-tokens 7` (llama.cpp's fixed depth 4 is
|
||
gone), greedy, `--think auto --preserve-thinking on`, text-only.
|
||
- **Client-contract deltas, all in the runbook:** the chat template is
|
||
compiled into the binary (no `--chat-template-file`, so the Qwen-Sharp
|
||
73k-token-prompt failure mode cannot recur); `reasoning_effort` now
|
||
accepts the full OpenAI set, so `models.yml`'s `reasoningEffortMap` fold
|
||
is no longer required — **kept anyway** so this commit does not also move
|
||
every `high` role from `xhigh` to `high` effort; and there is no
|
||
KV-quantization or prompt-cache-RAM flag.
|
||
- **Measured at cutover:** 13 s from `load_started` to listening (target
|
||
8 s, drafter 2 s, session 3 s) against a 20-minute `LLM_WAIT` budget;
|
||
RSS 30.6 GiB; a live short-prompt request returned
|
||
`Server-Timing: ttft;dur=549.8, inter_token;dur=24.2` (~41 tok/s).
|
||
Verified live: `/health` 200, `/ready` 200, `/v1/models` →
|
||
`frank-large` at `context_length 262144`, a chat completion carrying
|
||
`reasoning_content`, a `tool_calls` response with
|
||
`finish_reason: tool_calls`, and SSE streaming.
|
||
- **The trade, stated plainly.** gufo's own Q8_K_XL tables against
|
||
llama.cpp b11069: prefill +37 % to +80 %, cold load 4.5× faster, decode
|
||
+7 % at zero depth — but **−14 % at 65,536 tokens of depth and −24 % at
|
||
131,072**. On the 8-bit target the crossover is ~32k and frank-large runs
|
||
one 262,144-token conversation, so long agent sessions sit on the losing
|
||
side of it. The Q4_K_XL target shows no such regression; revisiting the
|
||
quantization is a separate decision.
|
||
- `/tests/frank_large_unit.bats` — Rewritten around behaviour: the seam
|
||
runs the serve script against a stub engine and asserts on the argv it
|
||
actually execs, instead of grepping the script's text. A caveat found
|
||
while writing it: **bats 1.13 ignores a failing `A && B` in the middle of
|
||
a test** (bash exempts the non-final element of an AND-list from
|
||
`errexit`), so every such assertion was split into standalone `[ ]` lines.
|
||
Mutations confirmed the seam bites: dead runtime gate, `DRAFT_TOKENS=0`,
|
||
a wrong context/sessions/port, an alias swap, a moved drafter, a removed
|
||
missing-asset message and a neutered commit pin each fail it.
|
||
- `/scripts/deploy-remote.sh` — `deploy_llm_large` gates on the runtime
|
||
binary as well as the payload and returns **before** it stops the serving
|
||
unit, so a host whose gufo was never built reports the failure instead of
|
||
taking the endpoint down on the way there.
|
||
- Docs: new `/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md`;
|
||
`frank-large-unsloth-qwen38-27b-q8xl.md` marked RETIRED (still the
|
||
rollback target — the prebuilt and the payload are both intact on disk);
|
||
AGENTS.md fleet table, frank-large section, bench policy and runbook
|
||
index; both vault indexes; `Dankclaw System Specs.md`; `README.md`.
|
||
- **e2e gate: 32/33.** The one failure is `192.168.0.112` (the FrankBook-Pro
|
||
Mac mini) at 100 % packet loss — offline, and the same failure recorded
|
||
in `564a781`. Not related to this change; no frank-large test failed.
|
||
- **Frontier-MoE feasibility study on Dankserver** (research + build, no
|
||
production change). Frank asked which of DeepSeek-V4.1-Flash,
|
||
MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash would run best on the 2x R9700 box,
|
||
and whether Affinity / vllm-radiance / FreeToken-AMD could be modified to run
|
||
them fast. Findings, all evidenced:
|
||
- **V4.1 does not fit.** 475.27 GiB checkpoint (measured from the HF API),
|
||
of which **189.13 GiB is Engram** in two 94.56 GiB shards. A token selects
|
||
240 routed experts at ~11 MB; at the 37-of-384-per-layer residency the hot
|
||
stack can hold, ~2.4 GB crosses PCIe per token. At 55 GB/s real that is a
|
||
**~23 tok/s ceiling** — below the 35 tok/s goal before any attention or
|
||
dense work. The 1000 t/s prefill target is ~20x away: the backend refuses
|
||
layer-major prefill for V4.1, so the only legal prefill is token-by-token.
|
||
- **Affinity** (Codeberg, StillDeadcode) is a *DeepSeek-V4-Flash* engine
|
||
claiming 110 tok/s on two R9700s; it **builds clean here** (hipcc 23,
|
||
gfx1201, zero errors). The 110 works because the 101 GiB container leaves
|
||
the cold expert set in a pinned host pool in RAM. A V4.1 port reuses the
|
||
quantiser, container, slab, host pool and placement engine (expert tensor
|
||
naming is identical, 15,360 tensors at 40x384) but must write CED, CSA2's
|
||
new indexer, the DSpark draft geometry and Engram — and Engram fails
|
||
*silently*, because V4.1 has no `num_hash_layers` key so Affinity's
|
||
`model.h:36` default of 3 survives. Weeks of work, same capacity wall.
|
||
- **Lucebox** is the only engine that runs V4.1 on gfx1201, on the unmerged
|
||
`feat/ds41-engine` branch. Its own DS41.md measures 0.8-1.9 tok/s on an
|
||
R9700 (best tuned three-tier case 9.1) with **61 GB of RAM**; Dankserver's
|
||
246 GiB is the one untested lever, so the 377.5 GiB ROCMFP23 payload is
|
||
downloading for a three-arm bench.
|
||
- `/serverAppConfigs-docs/LLM/Dankserver System Specs.md` — **RAM corrected
|
||
60 GiB -> 246 GiB** (stale since the 2026-09-20 board swap; the Lucebox
|
||
runbook had it right) and storage refreshed; `/mnt/entropy` is at 100%
|
||
(244 G) and must not be planned for payload.
|
||
- `/serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md` — new
|
||
runbook; indexed in `LLM/index.md`.
|
||
- **LANDMINE, not changed here:** `/home/dank/models` was cleared by Frank
|
||
mid-session to free space. frank-primary is still answering `:7770` from
|
||
mmapped-but-unlinked inodes (`/models` is empty inside the running
|
||
container) — **the endpoint cannot be restarted until the 175 GiB payload
|
||
is re-fetched.** A reboot or a container restart takes frank-primary down.
|
||
- Deploy-state reconciliation (Frank's call: no restart) — the autoresearch trio
|
||
(`f187165` PLE 1800, `41e9189` prefix caching, `ba2100e` docs) was found already
|
||
MERGED to main + pushed AND already live on frank-primary: the running
|
||
`frank-primary-r9v` container (booted 12:13 from the `ba2100e` checkout) carries
|
||
`--enable-prefix-caching` in argv and `VLLM_PLE_OFFLOAD_READY_TIMEOUT=1800` in env,
|
||
`/health` 200 — but `deploy-state/frank-primary` was still stamped `f187165`. A
|
||
`deploy.sh` run would have re-selected frank-primary and reloaded the 175 GiB
|
||
payload (~16–19 min) for zero behavioral change, on the endpoint this harness itself
|
||
runs on. Stamped the marker `f187165 → ba2100e` (status `deployed`, atomic
|
||
mktemp+mv per the deployer idiom); service untouched. e2e gate 32/33 — the lone
|
||
failure `11-admin` is the FrankBook-Pro Mac mini (`192.168.0.112:8989`) offline
|
||
(100 % ICMP loss + TCP timeout), the documented asleep-host condition, not a
|
||
regression.
|
||
- `scripts/frank-primary.env`, runbook — **prefix caching DEPLOYED on
|
||
frank-primary** (Frank's go-ahead; commit `41e9189`): E8-qualified, zero
|
||
VRAM cost, 9x/63x reuse TTFT. Verified on :7770 (argv flag, health 200,
|
||
completion "PREFIX-CACHE-LIVE"). Gotcha re-confirmed: the container is
|
||
in dank's ROOTLESS podman store (`sudo podman` inspects an empty store
|
||
and reports nothing); probe via the shim's `docker` or the unit MainPID.
|
||
- `/scripts/frank-primary.env`, `/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md`, `/serverAppConfigs-docs/index.md` — **frank-primary prefill/decode bench matrix** (scratch :7773, prod stopped 00:06 for the window per fleet bench law): baseline E0 (chunk 4096 / 80-slot sync / MTP4) = PP8K 954.6, PP131K 883.0 tok/s, decode 141.9 ms/step ≈ 23.8 t/s effective. Chunk 8192 (E1R): PP8K +25.9% (1201.4) but decode −8.5% and PP131K −4.4% — short-prompt win, decode tax, not the fleet's dominant shape. Chunk 16384 boots but OOMs at request time (0.93 GiB free) — 4096 is the ceiling (engine spec-config cap + memory). Expert cache 80 slots is the policy max (doctor: capacities 48/64/80, synchronous LRU only — async unsupported, 96/112 rejected); MTP depth 6 is 3.7x SLOWER than depth 4. Prefix caching (E8/E9): zero VRAM cost, cold path unchanged, reuse hits 32k 44.2→3.6–4.3 s (9x), 131k 439→6.9 s (63x); hit pattern rounds 1–2 miss / 3+ hit — RECOMMENDED, pending Frank's go-ahead. Two failure laws recorded: unqualified shapes fail mid-request (health stays 200), and PLE registration can blow the engine's 600 s `VLLM_PLE_OFFLOAD_READY_TIMEOUT` under page-cache churn (measured 740 s) — launch.sh gained `R9V_PLE_READY_TIMEOUT` forwarding (fork `2de0357`, local-only) and prod now sets 1800.
|
||
- **Incident — frank-primary down 00:06 → ~11:00 (11h)**: the bench window's required prod stop was never restored — the matrix + follow-ups ran 00:44 → 09:02 and the session let the endpoint sit dead (the 2026-09-18 "a bench session owes the fleet a health 200" lesson, re-offended). The first restore boot (09:46) then failed on the PLE 600 s timeout above (unit "Deactivated successfully", no journal error — the serve script exits 0 on the wait). Restored by the `R9V_PLE_READY_TIMEOUT=1800` fix deploy (f187165, from main — the deploy gate refuses non-main): health 200 + MainPID = the serve script's own `podman wait` + live completion verified ~11:00. Deploy e2e gate 32/33 — the one failure (11-admin) needs the FrankBook-Pro Mac mini, which was asleep and ignored two WoL bursts (240 packets); recurring pattern per the wol-wake skill. Lesson: a multi-hour bench window needs an explicit restore task, not a "watch for DONE" — DONE ≠ prod up.
|
||
|
||
## 2026-09-27
|
||
- `/scripts/frank-primary.env`, `/scripts/frank-primary.service`, `/scripts/serve-frank-primary.sh`, `/scripts/deploy-remote.sh`, `/tests/frank_primary_unit.bats`, `/serverAppConfigs-docs/LLM/frank-primary-r9v-radiance-qwen38-q8.md` (new), AGENTS.md + LLM index — **frank-primary cut over to the Q8_0 payload** (uncensored IQ4_XS → unsloth Qwen3.8-Flash-Next Q8_0, 175.30 GiB / 6 shards) at Frank's direction. Launcher gained the opt-in `R9V_CHAT_TEMPLATE_FILE`/`R9V_CHAT_TEMPLATE_KWARGS` passthrough (fork commit `30d4a32`, local-only) — the Qwen-Sharp v22.4.0 template now serves, fleet parity with frank-large. Env points at `/home/dank/r9v-state/q8xl/setup.json` + the quantized-inject dev-overlay adapter (fork `363d606`) + `R9V_FULL_MUTABLE_CACHE=0`; shard gates extended to SHARD4/5/6 (unset-safe); seams recalibrated; ctx window 131072 (models.yml maxTokens 36864). Spec decode is MTP4 only — the fork's `speculative_config` is a single dict, no chaining, so llama.cpp's `draft-mtp,ngram-mod` pattern is NOT expressible (PLE n-gram rides along architecturally). Uncensored state + payload remain on disk as the rollback target. 346 tests green.
|
||
- Dankserver frank-primary slot (experiment window) — the `radiance-vllm-qwen4exp` fork qualified the unsloth `Qwen3.8-Flash-Next-GGUF` **Q8_0** payload (175.30 GiB, 6 split parts) on the dual R9700. Two launch blockers fixed loader-side in the fork: `runtimes/dev-overrides/qwen4_exp.py` dequantizes quantized `block_inject` shards in-stream (the Q8_0 payload quantizes BOTH fused-HC shards; IQ4_XS ships inject as F32 — the uniformity check rejected the mixed projection) and `R9V_FULL_MUTABLE_CACHE=0` passthrough for the 80-slot tiered cache (fork commits 363d606, 2fe6dbb, 50a79ec — GitHub remote, not pushed). Battery on :7773: text + reasoning channel, vision ("Red" on a red PNG), prose ≈11.0 tok/s wall, MTP4 mean acceptance length 3.19–5.00; PLE pinned residency proven (`mapping_bytes=54400261120`, `mapping_rss_bytes` == mapping size, both ranks required 108.8 GiB vs 140.9 available). Relaunch recipe saved at `dankserver:/home/dank/r9v-state/q8xl/START-COMMAND.txt`; record at fork `docs/qualification/qwen38-q8-0-dual-r9700-2026-09-27.md`.
|
||
- Dankserver frank-primary restore — uncensored UD-IQ4_XS back on `0.0.0.0:7770` as `frank-primary-r9v` from its frozen config (health 200 in ~60 s; LAN-path completion returns content + reasoning); `frank-primary.service` (llama.cpp) **runtime-masked** for the window (`/run/systemd/system/frank-primary.service → /dev/null` — the `/etc` relink symlink blocks a plain mask; self-clears at reboot, unit file untouched) and the container runs `restart=no`, so a reboot returns the repo-configured llama.cpp with no contention. Restore config preserved at `dankserver:/home/dank/r9v-state/uncensored/restore-config-20260927.env`; scratch in `dankserver:~/.AGENTTRASH/20260927-q8xl-bringup/`.
|
||
|
||
## 2026-09-25
|
||
- `/serverAppConfigs-docs/Dankserver/Jellyfin.md` — Jellyfin LDAP 401 resolved: the auth works via the modern `/Users/AuthenticateByName` endpoint (HTTP 200, alex authenticated, full plugin flow logged: service bind → search → ldapUid → user bind → token). The 401 was a test artifact — I had been probing the legacy `/Users/Auth/Password` route, which Jellyfin 12.1 removed (the `LegacyEmbyRouteRewriteMiddleware` is gone). The web UI uses the modern endpoint, so user logins are fine. No config change needed; the LDAP-Auth 23.0.0.0 plugin + authentik 2026.8.3 outpost are healthy.
|
||
- `/shared/homepage/services.json`, `/shared/homepage/docs.html` — Suggested uses added to the three LLM model descriptions (status cards + the `/docs` model guide + the "which one to pick" foot-note): frank-large → cybersecurity + debugging (use requires Frank's permission); frank-medium → the primary intelligent model (surpasses Claude Sonnet 5 per Artificial Analysis) for software engineering, sysadmin, instruction tasks; frank-small → the fast/cheap utility model (agentic search, RAG, research subagents, translation, roleplay, summarization). Deployed to `/srv/homepage`; bats 19/19 + non-Mac-mini e2e green (d3c8197).
|
||
- `/shared/homepage/{services.json,app.js,style.css,docs.html}` — Model cards reworked to a structured three-line shape: each LLM card now carries a `details` object (role / spec / use) rendered as a bold pale-green role, a bright-green spec line, and a dim "Use:" line (`app.js` `descHTML()` falls back to the plain `description` for non-LLM cards; new `.desc-role`/`.desc-spec`/`.desc-use` CSS + bare-mode reset). Roles made neutral (no nicknames): frank-large = deep reasoning, frank-medium = primary model, frank-small = fast utility model. The `/docs` model guide was also fixed — it had drifted to retired model info (frank-large listed as Qwen3.8-Flash-Next 87.3 GiB / 200k, frank-medium as FP8 ~31 GiB) — now Qwen3.8-27B UD-Q8_K_XL 31.5 GiB / 262k and MXFP4/W4A8 19.4 GiB, reordered large→medium→small. bats 30/30 + live browser render verified.
|
||
- Fleet-wide container update (dankclaw + dankserver) — every pullable image brought current. Pinned bumps: authentik `2026.8.0`→`2026.8.3` (server/worker/outpost), tautulli `v2.18.1-ls242`→`ls245`, calibre-web `0.6.27-ls397`→`ls402`, plex `1.43.3.10896`→`1.43.4.10903`; seerr already at `v3.4.1`. Local-build rebuilds (forced `--no-cache`, the base images had moved): caddy (xcaddy porkbun), alloy, loki, comfyui-rocm (ComfyUI re-cloned), hermes ×3 (fresh `:latest`, `23986f31`→`e1e87aec`). anubis already at latest stable `v1.27.0` (only pre-releases exist beyond). Verified: all 16 floating-tag services CURRENT (zero stale), every real container active, caddy edge serving (auth 302 / ai 200), 340 bats green. Excluded: frank-medium/radiance-vllm (custom LLM build).
|
||
- `/dankserver/{tautulli,calibre-web,plex}/quadlet/*.container`, `/dankclaw/authentik/quadlet/*.container` — version pins bumped; docs (Tautulli.md, Calibre-Web.md, Plex.md, Authentik.md + server/worker, index.md) and the dead `dankserver/compose.yaml` plex entry synced (0b1ea2e).
|
||
|
||
## 2026-09-22
|
||
- `/serverAppConfigs-docs/**` — OKF v0.1 frontmatter migration: `type`/`description`/`tags`/`timestamp` added to the 65 non-reserved pages lacking them (timestamps from each file's last git commit); `timestamp` added to `LLM/frank-medium-batch-ubatch.md` and `LLM/frank-small-benchmark.md`; unquoted YAML scalars (colons, `*arr`) fixed in 13 pre-existing frontmatter blocks; frontmatter stripped from the reserved `LLM/Pipeline Parallelism/index.md`; `log.md` restructured to the OKF format (date-grouped, newest first, one line per change) from 40 narrative entries.
|
||
- `/serverAppConfigs-docs/**` — Fact-check pass against scripts/AGENTS.md ground truth: 6 false "Active" banners fixed on retired runbooks, 96→114 GiB GTT in `Shared Memory Architecture Dankclaw.md`, `deployment.md` + `Architecture/Deployment Flow.md` rewritten to the Quadlet/deploy-plan.py truth, `Calibre.md` 9.9.0→9.15.0 + CVE-2025-7404 conflation (it is the Calibre-Web CVE), `Copyparty.md` v1.20.24, `RomM.md` 5.3.0 breaking change; `*arr` docs moved to a new `FrankBook-Pro/` section (git mv, link-safe); active runbook bodies written for frank-large (unsloth UD-Q8_K_XL) and frank-small (Gemma4-12B).
|
||
- `/serverAppConfigs-docs/Research/` — Three new research briefs (OKF): `Local TTS and Voice Cloning.md`, `AI Job Search and Resume Tools.md`, `Most Popular Self-Hosted AI Apps.md` (the 15 most popular self-hosted AI apps of Jun–Sep 2026, ranked by GitHub stars) + a self-contained dark-theme HTML report `popular-self-hosted-ai-apps.html`; upstream-version rows added to the ComfyUI/OpenWebUI/Hermes app docs (ComfyUI 0.37.0 native Qwen-Image 2.1, OpenWebUI 0.11.4 + `/api/v1/images/generations`, Hermes v0.21.4/CalVer v2026.9.21). All 336 tests pass; 136 pages, 0 unresolved wikilinks; docs-only deploy no-op.
|
||
|
||
## 2026-09-21
|
||
- `/dankserver/frank-medium/quadlet/frank-medium.container` — frank-medium cut back to the MXFP4 checkpoint (`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq`); `KV_DTYPE` became a quadlet tunable, set `auto` (unquantized bf16 KV) to test whether fp8 KV had been degrading W4A8 output quality; pool auto-sized to 486,464 tokens; vision re-verified; e2e 32/32 (a45f396).
|
||
- `/dankserver/calibre-server/quadlet/calibre-server.container` — New boot-persistent Quadlet: Calibre Content Server on 192.168.0.235:8084 (OPDS + web UI), HTTP Basic auth, LAN-only; desktop image 9.15.0, `--auth-mode=basic` pinned, internal port 8090.
|
||
- `/serverAppConfigs-docs/Dankserver/Calibre.md` — Content-server section rewritten from "not running" to the deployed shape; Calibre-Web → authentik feasibility answered (guard chain, not in-app OIDC).
|
||
- `/scripts/serve-frank-large.sh`, `/scripts/frank-large.env`, `/scripts/frank-large.service`, `/scripts/download-frank-large-unsloth.sh` — frank-large (dankclaw :7770) moved CIRU Orca → unsloth llama.cpp b11030 prebuilt serving `unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS` (87.3 GiB), 200k ctx, MTP depth 2 via explicit `-md`; the split parts ARE the payload — concatenating them trips `invalid split file name` (part 1's header advertises `file_count=3`); draft acceptance verified live 0.83–0.92.
|
||
- `/serverAppConfigs-docs/LLM/frank-large-unsloth-qwen38-flash-next.md` — New runbook; `frank-large-ciru-orca.md` marked RETIRED (rollback target); AGENTS.md + LLM index updated.
|
||
- `/shared/prometheus/prometheus.yml` — `node-dankclaw` target fixed to `host.containers.internal:9100` (netavark DNATs published ports only; the node-exporter's host bind was unreachable by construction — a fourth down job the 09-16 audit had not listed); `Infrastructure/Prometheus.md` targets table + `tests/observability_quadlet.bats` contract updated.
|
||
|
||
## 2026-09-20
|
||
- `/dankserver/frank-medium/quadlet/frank-medium.container` — Checkpoint swapped MXFP4 → FP8 (`orcarouter/Qwen3.8-27B-Uncensored-FP8`, MTP self-speculation depth 8, acceptance 5.66/8); the swap is now one command via `scripts/hf-manifest.py` + `scripts/hf-fetch.sh` + `scripts/swap-frank-medium.sh`; A/B on the FP8 target: DFlash2 100.0 t/s decode beats MTP 85.9 (accept length is a proxy, not the metric) — DFlash2 is the deployed shape.
|
||
- `/dankserver/frank-medium/quadlet/frank-medium.container` — Cutover to vllm-radiance TP=2 (95.8 t/s / 0.21 s TTFT / 259.4 burst-4 vs llama layer split 45.7 / 0.72 / 71.7; the winning arm is also 4-bit, part of the margin is quant); the llama root unit moved to `scripts/.AGENTTRASH/20260920-llama-layersplit/` (never `systemctl disable`); KV pool auto-sized to 943,581 tokens (3.59× the window); tools, reasoning and vision verified live against the endpoint.
|
||
- `/scripts/build-llama-rdna4.sh` — Now takes `GGML_HIP_RCCL`; the RCCL build (`build-rdna4-rocm714-rccl`) made the tensor-mode bench arm fair (Ubuntu RCCL 2.27 carries gfx1201 device code; TheRock's does not — `invalid device function`).
|
||
- `/shared/prometheus/rules/llm-fleet.yml`, `/shared/prometheus/prometheus.yml`, `/shared/grafana/provisioning/` — Monitoring spans the `vllm:` + `llamacpp:` namespaces (6 alerts + 3 recording rules); frank-small/frank-large targets fixed to `host.containers.internal`; the prefix-cache reuse rule divides by `cached + prompt` (llama.cpp's `prompt_tokens_total` EXCLUDES cached tokens); prefix caching measured at 1,648-token block granularity (13,021-token prompt 2.30 s → 0.68 s, 76 % reused).
|
||
- `/scripts/dankserver-route-primary.sh` — `eno1` pinned primary (metric 100), `eno2` retained failover (metric 200); applied with `nmcli device reapply` (no `connection reapply` in NM 1.54).
|
||
- Hardware — dankserver board swapped to the ASUS ProArt B850-Creator WiFi NEO: both R9700s on CPU root ports at Gen5 x8, every bridge `AtomicOpsCap: Routing+`, both endpoints `ReqEn+` — the PCIe-atomics wall is gone; GPU BDFs moved, kernel 7.0.0-31, SMT back on (16 CPUs).
|
||
- `/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` — Now the ACTIVE runbook (deployed flags, battery, capabilities, rollback); `frank-medium-llamacpp-dflash2.md` carries the SUPERSEDED banner; `Dual-R9700-Cooperation.md` states the measured verdict; `Dankserver System Specs.md` gained a Motherboard section; `frank-medium-model-swap.md` is the new swap runbook; AGENTS.md topology/sections/lessons flipped.
|
||
|
||
## 2026-09-19
|
||
- `/scripts/frank-large.service`, `/scripts/serve-frank-large.sh` — frank-large: halogen → CIRU v4.4.1 Orca (jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca w4b, 262144 ctx, MTP depth 4, 4096 MiB PLE cache, text only); halogen is the rollback target (b6ce3af, f68bda5).
|
||
- `/serverAppConfigs-docs/LLM/frank-large-ciru-orca.md` — New runbook.
|
||
- `/scripts/frank-medium.env`, `/scripts/serve-frank-medium.sh` — frank-medium: ajgazin UD-Q8_K_XL → orcarouter Q8_0 + z-lab DFlash2 Q4_K_M drafter; split re-balanced 35:30 → 32:33; pool 393216 → 327680 (d374ab9).
|
||
- `/scripts/frank-small.env`, `/scripts/serve-frank-small.sh` — frank-small: same-day vLLM detour reverted to mainline llama.cpp Gemma4-12B Q4_K_M + MTP sidecar + ngram-mod, q8_0 KV, 4 slots of 262144 (374041c).
|
||
- Hardware — 3090 EVGA FTW3 iCX3 RGB set MODE_OFF and saved to EEPROM (persists across reboots, headless).
|
||
- AGENTS.md, both indexes, halogen runbook retirement banner (764cf41, 88331e2); 331 bats + 32 e2e green.
|
||
|
||
## 2026-09-18
|
||
- `/scripts/frank-medium.env` — `CACHE_RAM` 8192 → 16384: the old cap was binding (137 journal refusals; a full 196608-token slot state ≈ 12.8 GiB cannot fit 8 GiB); the new cap clears one full slot and stays under half the host RAM; `tests/frank_medium_unit.bats` seam added; `--ctx-checkpoints 0` confirmed a deliberate deviation (build default is 32) (1acca43).
|
||
- Incident — frank-medium found stopped twice: (morning) a bench round ended without the protocol's restore half; (midday) an orphaned `llama-server` from `third_party/llamacpp/build-merge` (PPID 1, loopback :17770) held ~16 GiB on each R9700 and would have starved the 35:30 allocation — proven abandoned (frozen CPU ticks, zero connections) then killed; health 200 in ~20 s. Lesson: "GPUs idle" is a check, not an assumption.
|
||
|
||
## 2026-09-17
|
||
- `/scripts/frank-small.env`, `/scripts/serve-frank-small.sh` — frank-small: ISTA-DASLab GSQ-RCO IQ3_S → `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` Q4_K_M + separate MTP assistant sidecar (`--spec-type draft-mtp,ngram-mod`); q8_0 KV (`build-cuda` rebuilt with `-DGGML_CUDA_FA_ALL_QUANTS=ON`, else ~12× prefill collapse); `--fit` chose the full 262144 window; audio transcribes verbatim; Gemma4 accepts every `reasoning_effort` level.
|
||
- `/scripts/frank-small.env` — `CTX_SIZE=1048576` / `PARALLEL=4` (four FULL 262144-token slots, the measured card ceiling: ~3,450 MiB/slot + ~8,546 fixed); ubatch 1024 deployed (pp ~965 / tg 226–302 at 253k; ubatch 2048 OOMs the MTP draft's pp buffer); draft n-max 4 + poll 100 measured WORSE at long ctx.
|
||
- `/serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` — New ACTIVE runbook; `frank-small-qwen38-gsq-rco.md` retired as rollback target.
|
||
- `/shared/homepage/` — The three LLM cards moved to the user page with live descriptions; new static `/docs` user page (access chain, model guide, how-tos) routed in the post-auth listener (32751c0).
|
||
- `/scripts/deploy-remote.sh` — The pending pass now prunes stale `pending` state entries for retired services into `~/.AGENTTRASH` and keeps going (a stale `frank-default-sightsound` entry made every `--pending` run exit 2) (02a7876).
|
||
- Fleet audit — Every vhost classified against the guard policy (no new exposure); `systemd-hermes` logged as a boot-persistence gap (running container, no quadlet); stray stopped `test-vllm` container on dankserver held for approval.
|
||
|
||
## 2026-09-16
|
||
- `/scripts/bench-batch-ubatch.sh` — New harness: one `llama-bench` invocation per cell (a test that cannot allocate `exit(1)`s, taking a whole sweep with it), per-cell VRAM sampler, service stopped/started from a trap; must run from a transient unit (`LimitMEMLOCK=infinity` is load-bearing) (87845fc, 77a07ff).
|
||
- `/scripts/frank-medium.env` — `UBATCH_SIZE` 1536 → 512 (prefill peaks at 512: 1193 vs 1078 t/s at 2048; 3.71 GiB cheaper) and `TENSOR_SPLIT` 36,29 → 35,30 (ubatch buffers cost ROCm1 ~2× what they cost ROCm0) (a11fae0); `tests/frank_medium_unit.bats` balance seam re-fitted as a shape assertion.
|
||
- `/serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md` — New page: the sweep, the logical-batch irrelevance, the amdgpu settle-after-kill trap.
|
||
- `/dankclaw/frank-large/quadlet/frank-large.container` — halogen restored as frank-large, retuned: ctx 262144 → 200192, pool 401408 → 400384, slots 3 → 2, max-tok 32768 → 24576, host-reserve 15 → 20 GiB, vision tower unset (moved to frank-small); ledger 95.4 GiB held / 14.7 GiB spare, no clamp.
|
||
- `/scripts/frank-small.env`, `/scripts/frank-small.service`, `/scripts/serve-frank-small.sh` — New frank-small: mainline llama.cpp 37b53fd on the RTX 3090 serving ISTA-DASLab GSQ-RCO IQ3_S with the in-file MTP head (`draft-mtp`), BF16 projector, 131072 ctx, f16 KV, port 7771.
|
||
- `/scripts/deploy-remote.sh` — `deploy_llm_large` flipped back to halogen with a `wait_port_free 7770` gate (the first cutover died on `Address already in use` two seconds after the old server exited); `deploy_llm_small` added.
|
||
- `/serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md`, `/serverAppConfigs-docs/LLM/halogen-flash-server.md` — New / re-activated runbooks; `frank-large-ik-uncensored.md` marked retired.
|
||
- `/serverAppConfigs-docs/LLM/x4-pair-experiment.md` — New page: 3090 + Strix Halo over the x4 link. iGPU-only Vulkan matches halogen's ~25 tok/s with the 3090 idle but commits 116 of 121 GiB (throughput parity, memory bankruptcy); per-tensor cross-device placement is killed by `-ot` itself (`has_tensor_overrides()` → `pipeline_parallel` → `n_copies == 1`); the first "CUDA + Vulkan cannot serve one model" conclusion was a llama-bench parsing artifact (`-dev A,B` is a SWEEP, not a list — correct `-dev A/B -ts x/y` gives 27.55 / 27.23 tok/s, +7 % decode / +16 % prefill vs iGPU-only).
|
||
- Incident — The rootless podman pause process died (trigger NOT established; no OOM kill in the kernel log): every `podman` command failed, all 28 quadlets looped, `podman system migrate` panics in 5.7.0; a reboot repaired it. The reboot exposed `hermes.franksdank.cloud` 502: the Caddyfile dialed `192.168.0.143:8787` (a host bind, not a published port) — fixed to `host.containers.internal:8787`, and the seam that had accepted any numeric upstream was strengthened.
|
||
- Host — Model cache cleared on operator instruction (298 → 895 GB free) and re-fetched: halogen's sidecars came back LARGER (in-place versioning — a stale copy is invisible from the file name); Q4_K_S vs Q4_K_M measured +2–3 % with ~5 GiB more host RAM; IQ4_XS fetched as the next comparison payload; `halogen-flash-server.md` gained a "Refreshing the weights" note.
|
||
|
||
## 2026-09-15
|
||
- `/scripts/frank-medium.env`, `/scripts/serve-frank-medium.sh`, `/scripts/frank-medium.service` — frank-medium: the 3090 left Dankserver, two R9700s (gfx1201) in; runtime charlie12345/ROCmFPX c49ebdb with its self-contained ROCm 7.14 toolchain (bypassing the host's broken mixed ROCm); kingjones Q8_0 AGENT + Q4_0 MTP head; layer split chosen by measurement (1991/18.8 t/s vs tensor 271.8/26.3 — tensor's decode win costs a 7× prefill collapse with no RCCL); draft pinned to ROCm1; frank-small retired (`retire_root_llm_small()` archived the CUDA-era unit).
|
||
- `/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md` — New runbook; `freetoken-frank-small.md` and `kingjones-q8agent-rocmfpx.md` marked retired/superseded; AGENTS.md, indexes, System Specs, deployment/Hermes/OpenWebUI/Plex/Stack pages updated.
|
||
- Host — Purged 167 stale AMD-repo ROCm 6.3.3 packages, removed the radeon apt pin and the `ld.so.conf` hijack (staged in `~/.AGENTTRASH/20260915-rocm633-cleanup/`).
|
||
- `/scripts/frank-large.service`, `/scripts/serve-frank-large.sh` — frank-large cut to ik_llama.cpp (halogen quadlet retired to `~/.AGENTTRASH/20260915-ik-swap/`); overnight decode campaign: 3.8–4.3 t/s across every safe configuration — the wall is ~100 CPU↔GPU section launches per token, not compute; mainline 37b53fd runs 1.0 t/s; the GGUF has no MTP head; 84 GiB of experts cannot fit 24 GiB VRAM (0c1e10a, 795b0b3).
|
||
- `/serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md` — New runbook with the full evidence chain.
|
||
|
||
## 2026-09-14
|
||
- `/dankclaw/compose.yaml`, `/dankclaw/.env.example` — New `hermes-webui` service (Hermex): `bootstrap.py --foreground` in the agent image, shares `/home/dank/hermes-data`, loopback-only publish 127.0.0.1:8787, password-gated via `HERMES_WEBUI_PASSWORD`.
|
||
- `/scripts/deploy-plan.py`, `/scripts/deploy.sh`, `/scripts/deploy-remote.sh` — `hermes-webui` registered as an isolated Dankclaw service in all three inventories.
|
||
- `/tests/hermes_webui.bats` — New seam (entrypoint, loopback publish + auth, shared volume, healthcheck, planner narrowing, deployer dispatch).
|
||
- `/serverAppConfigs-docs/Dankclaw/Hermes-WebUI.md` — New page; remote access = Termius local forward over the existing router :22 port forward.
|
||
|
||
## 2026-09-12
|
||
- `/dankclaw/marreta/quadlet/` — New Marreta content-cleaning proxy (Laravel) + Lightpanda browser quadlet pair, both joining gluetun's netns so browser fetch egress exits the ProtonVPN tunnel; `NGINX_HTTP_PORT=8085` (SearXNG holds 8080), gluetun publishes 8082:8085; guard chain marreta.franksdank.cloud → Anubis → authentik (91e9410).
|
||
- `/tests/marreta_quadlet.bats` — New seam (17 contracts); deploy-plan.py: marreta joins the gluetun netns family.
|
||
- `/serverAppConfigs-docs/Dankclaw/Marreta.md` — New page (incl. the `base64:`-prefixed APP_KEY gotcha).
|
||
- Live — authentik ProxyProvider `marreta` (pk 13) + Application link: the outpost's config API filters `application__isnull=False`, so a provider with no linked Application is never loaded (the login-wall root cause); a null `invalidation_flow` crashed the Rust outpost's serde and took down EVERY provider on the outpost; regression spec `16-marreta.spec.ts`.
|
||
|
||
## 2026-09-04
|
||
- `/frankbook-pro/` — *arr fleet consolidated on FrankBook-Pro as the sole deployment (dankserver compose copy removed, `/opt` config roots trashed); all six containers share gluetun's netns — all egress exits the ProtonVPN tunnel by construction; the arr UI ports moved onto the gluetun unit; qB download clients repointed `10.88.0.1:8080` → `127.0.0.1:8080`.
|
||
- `/tests/frankbook_quadlet.bats` — New seam: every arr+qB unit joins gluetun's netns and publishes no port; gluetun publishes the fleet's whole port set.
|
||
- `/dankserver/` — *arr stack reactivated behind six own-auth TLS vhosts (the root-owned `/opt/<svc>/config` dirs that crashed every app at logger init are gone); documented the verified rootless identity model: container root (PUID=0) maps to host uid 1000 by default, and `userns_mode` must NEVER be added (podman-compose runs services in pods; `--userns` + `--pod` is rejected).
|
||
- `/dankclaw/Caddyfile`, `/dankclaw/pihole/addn-hosts` — Sonarr moved to the admin side: the unguarded public vhost + post-auth handler retired, Pi-hole A record dropped — no WAN path at all; homepage card repointed at 192.168.0.112:8989.
|
||
- `/tests/e2e/` — Specs 11-admin/12-crawl de-hardcoded (expected open-link counts derive from the live services.json registry).
|
||
|
||
## 2026-09-03
|
||
- `/serverAppConfigs-docs/` — Full fact-check sweep of every page against the live hosts (read-only SSH), quadlet units, and the Caddyfile; surgical edits only (docs-only change = deploy no-op): Dankserver core rewritten to live quadlet facts, the Caddy guard chain documented end-to-end, Gitea data roots + UID gotcha corrected, `Prowlarr.md`/`prowlarr.md` deduped into one canonical page, archive LLM pages reworded from current-tense to dated historical framing.
|
||
|
||
## 2026-09-01
|
||
- `/frankbook-pro/gluetun/`, `/frankbook-pro/qbittorrent/` — WireGuard credentials provisioned (second Proton config, PA-12; one config = one keypair — two hosts on one config bounce the server-side endpoint and kill SearXNG); egress verified on the Proton exit; qB first-seed bug fixed (`WebUI\Password_PBKDF2`, not the legacy `WebUI\Password` key qB ≥ 4.2 ignores); `DefaultSavePath` → `/Volumes/COMPLETE/Finished Torrents`; download clients created in all four arrs; Sintel e2e through the tunnel at ~19 MB/s.
|
||
- `/serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md` — Rewritten to the active-tunnel reality (VM placement, WG gate, socat LaunchAgents, PBKDF2 seed); RDAP proved the "custom WG server" is a ProtonVPN Seattle server (full 65535-port scan: no admin surface anywhere).
|
||
- `/dankclaw/authentik/quadlet/authentik-outpost-ldap.container` — New LDAP outpost for Jellyfin (amd64 single-platform digest-pinned — podman 5.7.0 refuses the tag's nested manifest list); authentik objects provisioned via `ak shell` ORM; live bind tests proved two rules: the token intent must be `verification`, and the provider needs the `search_full_directory` object permission.
|
||
- `/serverAppConfigs-docs/Infrastructure/Authentik.md`, `/serverAppConfigs-docs/Dankclaw/Jellyfin.md` — LDAP-outpost section (objects table + rules) and the LDAP-Auth 23.0.0.0 plugin section (`LdapSearchAttributes` is comma-separated; username attribute `cn`).
|
||
- `/scripts/frank-small.env`, `/scripts/serve-frank-small.sh` — Two compressed-tensors FP8 swaps attempted on FreeToken and both failed to load (ornith-1.5: expert naming; HauhauCS aggressive: block scales do not broadcast in `_dequant_fp8_weight`); reverted to Huihui FP8 within the hour; `freetoken-frank-small.md` quant matrix updated (modelopt block-fp8 is the only FP8 path that loads); the deploy health gate passed for BOTH failed loads — a swap is only proven by the journal + a real completion.
|
||
- `/dankserver/{romm,jellyfin,calibre-web,copyparty}/quadlet/` — After a hard crash left every bare compose container dead at boot, the media stack standardized to quadlet: six units shipped + cutover handlers (retire bare containers, rescue RomM's anonymous volume, merge-move `/opt/<svc>` → `/home/dank/Containers/<svc>`); v1 generator quirks fixed (`Command=`→`Exec=`, `Device=`→`AddDevice=`, `HealthCmd=` space-splitting, the `HOME=/root` EnvironmentFile trap).
|
||
- `/dankclaw/` — Data roots of gitea, authentik, and the hermes trios moved to `/home/dank/Containers/<svc>/` while running (rename is inode-bound — zero downtime); `deploy_hermes` ordering fixed (generated units reload AFTER the unit files are copied).
|
||
- `/dankclaw/seerr/quadlet/seerr.container` — New Seerr v3.4.1 request manager: loopback-only 127.0.0.1:5055, guarded vhost + outpost provider `seerr-proxy`; `/tests/seerr_quadlet.bats` new seam; `/serverAppConfigs-docs/Dankclaw/Seerr.md` new page.
|
||
- `/scripts/deploy-remote.sh` — frankbook handler: the credential-gate SKIP state now propagates Mac-side before restart+verify; qB first-boot seed writes a PBKDF2 password + `HostHeaderValidation=false` + the four categories; full 7-service deploy run.
|
||
- `/frankbook-pro/qbittorrent/quadlet/qbittorrent.container` — qB pinned `4.6.7` (5.x's `204` login breaks Readarr's retired client; 5.x's conf rewrite leaves a PBKDF2 line 4.6.7 rejects).
|
||
- `/serverAppConfigs-docs/Dankserver/{RomM,Jellyfin,Calibre-Web,Copyparty}.md`, `/serverAppConfigs-docs/Architecture/Storage Policy.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — Updated to the quadlet reality, the Containers layout, and the two ownership regimes.
|
||
|
||
## 2026-08-30
|
||
- `/scripts/frank-npu.env`, `/scripts/serve-frank-npu.sh`, `/scripts/frank-npu.service` — New `frank-npu` on Dankclaw: FLM v1.0.1 serving the whole local registry (36 tags) on 52626; first deploy crash-looped (`/dev/accel/accel0` err=-19) — root cause: the XDNA2 SVA bind fails whenever the AMD IOMMU is off; fixed with `iommu=pt` + reboot; vision + audio towers proven end-to-end; health probe is `/v1/version` (FLM has no `/health` route) (c3ed3a6, cf0c541).
|
||
- `/serverAppConfigs-docs/LLM/frank-npu.md` — New runbook; AGENTS.md fleet table + the IOMMU/SVA hard requirement.
|
||
- `/dankclaw/frank-large/quadlet/frank-large.container` — frank-large: swapped to `kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF` on the pinned `kingjones30/ROCmFPX` fork (fa-on works UNPATCHED — the QSA F32 assert was agentionai-specific); ~225 s load, real 111k-token prefill at 213–232 t/s.
|
||
- `/scripts/frank-small.env` — frank-small (FreeToken): model swapped to `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` (native block-fp8); ~40 tok/s verified.
|
||
- `/serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md` — Reactivated as the large runbook; `agentionai-fast-imatrix.md` marked RETIRED; `freetoken-frank-small.md` model sections updated; LLM index + AGENTS fleet tables refreshed.
|
||
- `/serverAppConfigs-docs/Dankserver/prowlarr.md`, `/serverAppConfigs-docs/Dankserver/qbittorrent.md` — Rewritten: the arr↔Prowlarr feed contract (`/{indexerId}/api`, the global key — the hand-rolled `/torznab/` paths were Forms-auth catch-alls returning 302), TV + music chains proven live, first real grab (The Dark Knight 2160p, imported), Prowlarr auto-sync broken by version skew, qB persistent credentials (`WebUI\Password_PBKDF2`, host-local 0600), Readarr parked (abandoned upstream; its client cannot authenticate against qB 5).
|
||
|
||
## 2026-08-29
|
||
- `/dankclaw/Caddyfile` — `:8016` is now a uniform whole-request `reverse_proxy authentik-server:9000` preserving `Host`; the per-vhost redirect/flow design and stale comments deleted (29f9a86).
|
||
- `/dankclaw/anubis/policy.json` — ALLOW `^/outpost\.goauthentik\.io/` so login/transport paths are reachable before the challenge dance (2a72efa); the policy bind-mount pins the file inode — a git checkout does NOT reach the running distroless container, so a `--service anubis` deploy restart is mandatory after policy edits.
|
||
- `/tests/authentik.bats` — Meta-header test replaced by the upheld contract (Host preserved; no absolute cross-host `next=` anywhere in the Caddyfile).
|
||
- `/serverAppConfigs-docs/Infrastructure/Authentik.md`, `/serverAppConfigs-docs/Infrastructure/Anubis.md`, `/serverAppConfigs-docs/Architecture/Network Architecture.md` — Rewritten to the live-verified mechanism: the 2026.8 embedded outpost routes solely on the original `Host` vs each provider's `external_host` (meta-header routing disproven live); proxy providers + applications provisioned for search/hermes/prometheus/romm/files; null `authorization_flow`/`invalidation_flow` starve the outpost's config fetch; the public zone gained the `auth` A record via the Porkbun API (ee1e038).
|
||
|
||
## 2026-08-26
|
||
- `/dankclaw/Caddyfile` — Both Plex vhosts proxy to PMS over a TLS upstream (`insecure_skip_verify`); PMS drops plain-HTTP connections from non-loopback sources (empty-body 502s).
|
||
- `/dankclaw/gitea/quadlet/gitea-caddy.container` — Standalone Gitea Caddy quadlet retired; web proxy ownership moved to canonical Caddy; deploy-plan/deploy.sh/deploy-remote.sh + tests updated for the migration.
|
||
- Incident — Post-reboot: Caddy (a bare compose container — `restart: unless-stopped` does not survive shutdown's explicit stop) and Pi-hole (quadlet crash-loop, restart counter 1645) were both down. Root cause: `net.ipv4.ip_unprivileged_port_start` reset to 80 by a later-sorted sysctl drop-in AND a stale line in `/etc/sysctl.conf`, which systemd-sysctl applies AFTER all of `sysctl.d/`.
|
||
- `/AGENTS.md` — New MANDATORY "Boot Persistence & Self-Healing" section.
|
||
- `/dankclaw/caddy/quadlet/caddy.container`, `/dankclaw/caddy/env.example` — Caddy is now a boot-persistent quadlet (joins `dankclaw_default`, `Pull=never`, env from `/home/dank/Containers/caddy/env`); removed from compose.yaml.
|
||
- `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — Repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` stages conflicting drop-ins, comments out the stale sysctl.conf line, and applies `sysctl --system` (the boot code path).
|
||
- `/tests/caddy_quadlet.bats` (new) + caddyfile/deployment_flow/pihole seams — quadlet boot persistence, self-healing, network, env, and sysctl coverage.
|
||
- `/serverAppConfigs-docs/Infrastructure/Caddy.md`, `/serverAppConfigs-docs/Infrastructure/Pi-hole.md`, `/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md`, `/serverAppConfigs-docs/Dankclaw/Gitea.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — quadlet runtime facts, the 2026-08-26 incident, the sysctl-conflict failure mode, canonical Caddy ownership.
|