Frank
d8dbd4f5bc
Knowledge-base follow-up to a45f396. The runbook, both index one-liners, and
the AGENTS.md topology row all still declared FP8 active; flipped to
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq with unquantized bf16 KV (pool
486,464 tokens, full 262,144 window), and recorded the causal nuance: fp8 KV
was on both arms, so the FP8-vs-MXFP4 gap was weight precision, not KV - clean
KV is a quality hypothesis for W4A8, not a throughput one. No verdict yet.
The quadlet's "TEXT ONLY ... 400 here" comment was wrong: vision works on the
MXFP4 checkpoint (Qwen3_5ForConditionalGeneration carries the vision tower
through the Gemma4 merge path, unquantized) - verified by naming a solid red
48x48 PNG "Red". Comment corrected; harness models.yml input stays [text,image].
Docs-only; no re-deploy (the endpoint already serves vision).
1400 lines
103 KiB
Markdown
1400 lines
103 KiB
Markdown
---
|
||
type: reference
|
||
title: Bundle Change Log
|
||
description: Chronological record of meaningful changes to the serverAppConfigs knowledge base.
|
||
tags:
|
||
- changelog
|
||
- documentation
|
||
- maintenance
|
||
timestamp: 2026-09-14
|
||
---
|
||
|
||
# Bundle Change Log
|
||
|
||
## 2026-09-21 — frank-medium cut back to MXFP4 with unquantized bf16 KV
|
||
|
||
- **The change.** frank-medium (dankserver :7770) served the FP8 checkpoint
|
||
(`orcarouter/Qwen3.8-27B-Uncensored-FP8`) with fp8 KV from 2026-09-20 to
|
||
09-21; it is now cut back to the MXFP4 checkpoint
|
||
(`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq`, quark/W4A8) with
|
||
`--kv-cache-dtype auto` — unquantized bf16 KV instead of the endpoint's
|
||
historical fp8. `KV_DTYPE` is now a quadlet tunable (entrypoint default fp8);
|
||
`MAXLEN` stays at the native 262,144. Commit a45f396.
|
||
- **Why.** The operator's read that fp8 KV was hurting MXFP4. The nuance: fp8
|
||
KV sat on BOTH the FP8 and MXFP4 arms (hardcoded on the common exec line in
|
||
every commit since the MXFP4 era), so it could not explain the FP8-vs-MXFP4
|
||
*throughput* gap — that was weight precision. But MXFP4 is W4A8, and fp8 KV
|
||
on top compounds attention-path precision loss; clean KV tests whether that
|
||
was degrading *output quality*. No verdict yet — this is the setup.
|
||
- **Measured on deploy.** `kv_cache_dtype=auto`, no fp8 banner; pool auto-sized
|
||
to **486,464 tokens** (`Maximum concurrency for 262,144 tokens: 1`) — exactly
|
||
half MXFP4's 943,581-token fp8 pool (the 2x f16/fp8 ratio), and MXFP4's freed
|
||
weights (19.4 vs 30.9 GiB) keep the full 262,144 window fitting. Vision
|
||
re-verified on MXFP4 (a solid red 48x48 PNG named "Red"); the quadlet's old
|
||
"TEXT ONLY" comment was wrong and is corrected. e2e 32/32.
|
||
|
||
## 2026-09-21 — Calibre Content Server deployed (dankserver :8084)
|
||
|
||
- **The change.** The Calibre Content Server (`calibre-server`) is now a
|
||
live, boot-persistent, self-healing Quadlet service on dankserver
|
||
(`dankserver/calibre-server/quadlet/calibre-server.container`), serving
|
||
the shared library at **192.168.0.235:8084** (OPDS + web UI), HTTP Basic
|
||
auth, single admin `calibre`. LAN-only: no vhost, no public DNS.
|
||
- **Why the desktop image.** The compose-era snippet assumed the old
|
||
headless `lscr.io/linuxserver/calibre`; the current 9.15.0 is a desktop
|
||
build (Selkies GUI). The dedicated headless images (ta264, agentgreen)
|
||
are dormant (calibre 6.10 / 2017, both below the 9.9.0 content-server
|
||
fix). The unit runs `calibre-server` as the image's s6-overlay main
|
||
service via `Exec=`; the desktop also runs but its ports are
|
||
internal-only (pasta is port-forward mode here — only the 8084 forward
|
||
is exposed).
|
||
- **Auth traps found by testing.** (1) calibre 9.x users live in a SQLite
|
||
userdb whose `pw` column is plaintext (file must be mode 600); bootstrap
|
||
is `--manage-users -- add` with the password on stdin via `printf` — an
|
||
`echo` trailing newline lands in the stored password and breaks every
|
||
login. (2) On a non-SSL port the server defaults to Digest auth and
|
||
rejects Basic clients with `400 Unsupported authentication method` —
|
||
`--auth-mode=basic` is pinned. (3) 8080 inside the container is the
|
||
image's own nginx, so the content server runs on internal 8090.
|
||
- **Calibre-Web → authentik.** Feasibility answered: NOT via in-app OIDC
|
||
RP (Grafana's path) — Calibre-Web 0.6.27 has no generic OIDC (verified
|
||
in the container: no OIDC config keys, no authlib; upstream has an open
|
||
feature request and the maintainer declines more login options). The
|
||
fleet-standard path is the guard chain (Anubis → authentik proxy
|
||
provider), which needs no app change; Calibre-Web's own login (local /
|
||
LDAP / GitHub / Google only) would remain a second layer, romm-style.
|
||
Not migrated yet (verification-only per the operator).
|
||
- **Docs.** `Dankserver/Calibre.md` content-server section rewritten from
|
||
"not running" to the deployed shape.
|
||
|
||
## 2026-09-21 — frank-large: CIRU Orca → unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS
|
||
|
||
- **The change.** At the operator's request, frank-large (dankclaw :7770)
|
||
moves from the CIRU v4.4.1 Orca runtime (jcbtc
|
||
Qwen3.8-Flash-CIRU-STRIX-Orca, 262144 ctx) to the unsloth llama.cpp
|
||
prebuilt (b11030-mix-5ff778e, ROCm gfx1151, bundles its own ROCm 7.16)
|
||
serving `unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS` (87.3 GiB, three HF
|
||
split parts), one conversation at 200,000 ctx, MTP depth 2.
|
||
The fleet contract is unchanged: port 7770, alias `frank-large`,
|
||
text-only.
|
||
- **Why the unsloth prebuilt.** The qwen4exp MTP graph exists only in the
|
||
unsloth fork (stock mainline has no MTP for this architecture); the
|
||
prebuilt bundles its own ROCm, so there is no host SDK gate or venv the
|
||
way the CIRU build had one.
|
||
- **The explicit `-md`.** Sidecar auto-discovery does not search the `mtp/`
|
||
subfolder — without an explicit `-md <head>`, `--spec-type draft-mtp`
|
||
silently runs at the main model's speed. The serve script gates on the
|
||
head's presence and passes it.
|
||
- **Memory model.** The iGPU's budget is the GTT aperture: 116,732 MiB
|
||
(~114 GiB), `gtt_total=122402963456` — not the old "64 GiB rocm-smi"
|
||
lesson (stale: the CIRU Orca held 92.6 GiB on the same device). 200k
|
||
budget ≈ 100 GiB, ~14 GiB of headroom.
|
||
- **Verified before cutover.** Gemma4-12B smoke load on the iGPU via the
|
||
prebuilt: clean completion with reasoning extraction (the first probe
|
||
returned empty content because the thinking block consumed the 32-token
|
||
budget — a budget artifact, not a decode failure). CIRU restored for the
|
||
~3-hour model fetch (health 200 in ~40 s).
|
||
- **Docs.** New runbook `frank-large-unsloth-qwen38-flash-next.md`;
|
||
`frank-large-ciru-orca.md` marked RETIRED (rollback target, payload and
|
||
runtime intact on disk); AGENTS.md topology/section/runbook index and the
|
||
GPU-budget lesson updated; LLM index header + mermaid + active list
|
||
updated.
|
||
- **Files.** `scripts/serve-frank-large.sh`, `scripts/frank-large.env`,
|
||
`scripts/frank-large.service`, `scripts/download-frank-large-unsloth.sh`,
|
||
`scripts/deploy-remote.sh`, `tests/frank_large_unit.bats`.
|
||
- **The split-filename trap (caught at cutover).** The first cutover
|
||
crash-looped the unit ~110 times with `invalid split file name` at model
|
||
load (~50 ms, before any weights were read): the fetch script
|
||
concatenated the three parts into a single GGUF, but part 1's header
|
||
carries `ggml.file_count=3`, so the byte-correct concatenation still
|
||
advertises itself as a 3-way split and the loader demands a
|
||
`-NNNNN-of-NNNNN` filename. Fix: the parts ARE the canonical payload —
|
||
the serve script points `-m` at part 1 and llama.cpp loads the split
|
||
natively (parts 2–3 auto-discovered in the same directory). A manual
|
||
part-1 load confirmed the fix (model loaded in ~34 s from a warm page
|
||
cache, health 200) before the redeploy. The redundant 87 GiB assembled
|
||
file is staged at `~/.AGENTTRASH/20260921-frank-large-assembled/`
|
||
(dankclaw) awaiting purge approval.
|
||
- **Verified after cutover (live).** Unit active; the journal shows the
|
||
draft model loading and `model loaded` in ~37 s (warm page cache), then
|
||
`listening on http://0.0.0.0:7770`. The MTP head is genuinely attached
|
||
and drafting — `draft acceptance = 0.91667 (22/24), mean len = 2.83`
|
||
and `0.83333 (30/36), mean len = 2.67` (n_max 2, i.e. both drafts
|
||
accepted most of the time), not the silent main-model-speed mode.
|
||
`/v1/models` → `frank-large`; a chat completion returns `content` plus
|
||
extracted `reasoning_content`; a tool request returns a clean
|
||
`tool_calls` array (`get_weather {"city":"Seattle"}`). The e2e gate
|
||
failed once on the known-flaky `12-crawl` (an `app.plex.tv` link
|
||
timed out — unrelated to this endpoint) and passed 32/32 on re-run.
|
||
|
||
## 2026-09-21 — node-dankclaw Prometheus target fixed (netavark host-bind quirk)
|
||
|
||
- **The gap.** A deep audit found the `node-dankclaw` scrape target red while
|
||
its container reported healthy. node-exporter runs `Network=host`, so its
|
||
`192.168.0.143:9100` bind is a host bind, not a published port — and this
|
||
Prometheus is a bridge container (`Network=dankclaw_default`) that, on this
|
||
netavark host, can only reach the host's LAN IP at a PUBLISHED port. The
|
||
target was unreachable by construction, not the exporter down. This was a
|
||
fourth down job the 2026-09-16 "three jobs read down" audit did not list.
|
||
- **The fix.** Scrape `host.containers.internal:9100` instead — netavark
|
||
delivers that address to the host's LAN IP, where node-exporter listens.
|
||
The same rule `frank-large` and `frank-small` already use (documented in the
|
||
LLM comment block of `shared/prometheus/prometheus.yml`). `node-dankserver`
|
||
is a different machine and keeps its LAN address. Verified live: the
|
||
workaround returns real metrics from inside the container, and after deploy
|
||
`up{job="node-dankclaw"}` is 1 — the down set is back to exactly the three
|
||
structural jobs (comfyui, services, gitea).
|
||
- **Files.** `shared/prometheus/prometheus.yml` (target + comment),
|
||
`serverAppConfigs-docs/Infrastructure/Prometheus.md` (Targets table), and the
|
||
cross-file contract test `tests/observability_quadlet.bats` (now asserts the
|
||
exporter binds the LAN address AND the local job scrapes via the gateway,
|
||
instead of asserting scrape-target == LAN address, which the quirk breaks).
|
||
|
||
## 2026-09-20 — DFlash2 vs MTP A/B on the FP8 checkpoint: DFlash2 wins on decode
|
||
|
||
- **The test.** The FP8 checkpoint carries its own single-layer MTP head, so
|
||
it can self-speculate (`SPEC_METHOD=mtp`) or use the z-lab DFlash2 external
|
||
drafter (`SPEC_METHOD=dflash`). Both were deployed and measured on the same
|
||
FP8 target (`scripts/bench-conversation.py`, 10 real prompts, `max_tokens=512`,
|
||
greedy, salted, `--burst 4`):
|
||
- **DFlash2** (depth 7): mean accept 4.24/7, geomean decode **100.0 t/s**,
|
||
TTFT 0.19 s, prefill 923.0 t/s, burst-4 **287.1 t/s**, KV pool 384,238 (1.47×).
|
||
- **MTP** (depth 8): mean accept 5.66/8, geomean decode 85.9 t/s, TTFT 0.06 s,
|
||
prefill 1096.9 t/s, burst-4 179.7 t/s, KV pool 461,540 (1.76×).
|
||
- **Verdict: DFlash2 wins on decode (+16 %) and burst-4 (+60 %); MTP wins on
|
||
TTFT and prefill.** The counterintuitive part: DFlash2 accepts *fewer*
|
||
tokens per step (4.24 vs 5.66) yet decodes *faster*. Accept length is a
|
||
proxy, not the metric — throughput is `(accept + 1) / (draft time + verify
|
||
time)`. The DFlash2 drafter is a small separate model that steps fast; the
|
||
MTP head is the target's own layer applied autoregressively 8×, so its draft
|
||
is slower. The faster step more than makes up the lower accept length.
|
||
- **FP8 + DFlash2 (100.0 t/s) beats the original MXFP4 + DFlash2 (92.4 t/s)**
|
||
— the higher-precision FP8 target verifies more of the drafter's proposals.
|
||
The drafter costs 2.1 GiB of VRAM (KV pool 1.47× vs MTP's 1.76×), an
|
||
acceptable trade for the decode win.
|
||
- **DFlash2 is the deployed shape.** The quadlet comment, the radiance
|
||
runbook (deployed-flags table, KV pool, a new A/B section), AGENTS.md
|
||
(topology row + frank-medium section), and the model-swap runbook + swap
|
||
script (both paths now default to `dflash/7`) all updated.
|
||
|
||
## 2026-09-20 — frank-medium checkpoint swapped MXFP4 → FP8; the swap is now one command
|
||
|
||
- **The swap.** `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4,
|
||
19.37 GiB single shard, DFlash2 external drafter) →
|
||
`orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise 128×128,
|
||
dynamic activation quant, 30.9 GB / 7 shards, its own single-layer MTP head).
|
||
The quantization-aware entrypoint (`radiance-entrypoint.sh`) reads
|
||
`quant_method` from `config.json` and serves the plain FP8 path — no MXFP4
|
||
patches, `--quantization fp8`, `ROCM_AITER_UNIFIED_ATTN`, MTP
|
||
self-speculation (`SPEC_METHOD=mtp`, `SPEC=8`). The quadlet edit was four
|
||
env vars (`SNAP`, `SPEC_METHOD`, `SPEC`, `KV_MEM`); the deploy handler
|
||
(`deploy-remote.sh`) now reads the serving shape from the quadlet and gates
|
||
on the right assets (drafter only for `dflash`, `r4d.so` only for
|
||
`mxfp4`/`quark`).
|
||
- **MTP measured, not inferred.** Boot log `SpecDecoding metrics`: mean
|
||
acceptance length **5.66/8**, 66.2 % draft acceptance, per-position rates
|
||
0.89 → 0.41. The FP8 model carries its own single-layer MTP head (22
|
||
tensors), so self-speculation needs no external drafter and no `fp8_mtp.py`
|
||
rewrite.
|
||
- **KV pool auto-sized to 461,540 tokens (1.76× the window)** — smaller than
|
||
MXFP4's 943,581 (3.59×) because the FP8 weights are bigger (30.9 vs 19.4
|
||
GiB). `KV_MEM=0` (the documented plain-FP8 default); recalibrate with
|
||
`calibrate-kv.sh` after the first healthy start.
|
||
- **Battery (vs the MXFP4 baseline).** geomean decode 85.9 vs 92.4 t/s (−7 %),
|
||
mean TTFT 0.06 vs 0.22 s (3.7× faster), geomean prefill 1097 vs ~1490 t/s
|
||
(−27 %), burst-4 aggregate 179.7 vs 248.9 t/s (−28 %). The trade-off is the
|
||
expected one: higher precision (8-bit vs 4-bit weights), decode within 7 %,
|
||
TTFT much faster, prefill/burst slower (the `ROCM_AITER_UNIFIED_ATTN` path
|
||
vs R4D). Vision, tools, and reasoning all verified live.
|
||
- **Streamlining — the long pole was the download, not the GPU.** Three tools
|
||
make a swap one command: `scripts/hf-manifest.py` (generate a
|
||
`<filename> <size> <sha256>` manifest from the live repo tree — LFS sha256s
|
||
from the resolve endpoint's `X-Linked-ETag`, sizes from `Content-Length`),
|
||
`scripts/hf-fetch.sh` (resumable, byte + sha256 verified, idempotent, token
|
||
out-of-band via a mode-600 `curl -K` file), and
|
||
`scripts/swap-frank-medium.sh` (generate manifest → download → detect
|
||
quantization → edit the quadlet → deploy). A swap is now
|
||
`swap-frank-medium.sh --repo <owner/repo> --target /home/dank/models/<name>
|
||
--generate-manifest --apply --deploy`. Full process: [[frank-medium-model-swap]].
|
||
- **Docs.** [[frank-medium-radiance-mxfp4]] description + deployed-flags table
|
||
+ KV-pool note flipped to the FP8 shape (the MXFP4 checkpoint is the
|
||
rollback target); [[frank-medium-model-swap]] is the new swap runbook; the
|
||
LLM index's frank-medium entry flipped. The MXFP4 checkpoint stays on disk
|
||
at `/home/dank/models/Qwen3.8-27B-Uncensored-MXFP4-awq`.
|
||
|
||
## 2026-09-20 — frank-medium cut over to radiance TP=2; monitoring follows the runtime
|
||
|
||
- **Measured first, then cut over.** Three arms of `scripts/bench-conversation.py`
|
||
(10 real-workload prompts, `max_tokens=512`, greedy, salted per arm, service
|
||
stopped between arms per the Benchmarking Policy): llama.cpp layer split 32:33 +
|
||
DFlash2 Q4_K_M **45.7 t/s** decode geomean / 0.72 s TTFT / 71.7 t/s burst-4;
|
||
llama.cpp `--split-mode tensor` 1:1 drafterless **30.5** / 0.71 / 54.3;
|
||
vllm-radiance TP=2 + DFlash2-FP8 **95.8** / 0.21 / 259.4 (1493.6 t/s prefill).
|
||
Honest caveat recorded everywhere: the winning arm is also 4-bit where the
|
||
llama arms are 8-bit, so part of the margin is quantization, not parallelism.
|
||
- **RCCL built for the question.** `scripts/build-llama-rdna4.sh` takes
|
||
`GGML_HIP_RCCL` now; the RCCL-backed `build-rdna4-rocm714-rccl` boots llama.cpp
|
||
tensor mode with a real collective, which is what made arm L a fair test.
|
||
vLLM's own collective is R4D one-shot, not RCCL — TheRock's RCCL has no gfx1201
|
||
device code (`invalid device function`).
|
||
- **Deploy.** `deploy.sh dankserver --service frank-medium`: the llama.cpp root
|
||
unit's symlink moved to `~/.AGENTTRASH/20260920-llama-layersplit-frank-medium/`
|
||
(never `systemctl disable` — it deletes the file), the Quadlet installed and
|
||
enabled, boot in 181 s, `/health` green. Post-deploy re-measure reproduced the
|
||
bench: 92.4 t/s, 0.22 s TTFT, 248.9 t/s burst-4.
|
||
- **KV pool is auto-sized, not pinned.** Production ships no `KV_CACHE_MEMORY_MB`:
|
||
the pool came up at **943,581 tokens = 3.59× the 262,144 window**, container RSS
|
||
flat at 28.4 GiB through a 4-way burst, host swap unchanged at 3.3 GiB. The
|
||
18,563,072,000-byte pin was an arm-R measurement artifact (one full-context
|
||
sequence), not the deployed shape.
|
||
- **Capabilities verified against the live endpoint, not inferred**: `/v1/models`,
|
||
tool calls (`qwen3_coder`, harness drove a real bash call through it),
|
||
`reasoning_effort` none/low/medium/high with thinking in `reasoning`, drafter
|
||
accept length 2.9–4.3, and **vision works** despite `language_model_only: true`
|
||
in `config.json` — the model identified shape and colour on held-out synthetic
|
||
images. `~/.omp/agent/models.yml` therefore lists `input: [text, image]`,
|
||
`contextWindow: 262144`, `maxTokens: 32768` (compaction gap 183,500 + 32,768
|
||
≤ 262,144).
|
||
- **Monitoring cut over with the runtime.** `shared/prometheus/rules/llm-fleet.yml`
|
||
now spans two namespaces — `vllm:*` for frank-medium, `llamacpp:*` for the
|
||
Dankclaw pair — with 6 alerts + 3 recording rules; KV-saturation and preemption
|
||
pressure returned (vLLM exports both), and the two engine-independent rules OR
|
||
the families with per-family traffic floors. One semantics trap fixed the same
|
||
day: llama.cpp's `prompt_tokens_total` EXCLUDES cached tokens (its own HELP
|
||
line), so the reuse rule divides by `cached + prompt` — the naive
|
||
`cached/prompt` reads 77 on a fully-warm endpoint and can never show a
|
||
collapse. **Two scrape targets were red by construction and are fixed**:
|
||
frank-small and frank-large are *host processes* on Dankclaw and netavark DNATs
|
||
only published ports, so both are now addressed as `host.containers.internal`.
|
||
The provisioned Grafana dashboard was re-cut the same day: every panel reads
|
||
both namespaces (the endpoint picker selects), the KV-fill panel returned with
|
||
`vllm:kv_cache_usage_perc`, the latency p95 panel returned with the vLLM
|
||
histograms, and the alerts panel is scoped to the fleet's `Llm.*` rules. Every
|
||
rule and dashboard expression validated against the live endpoints. **Prefix
|
||
caching measured, not inferred**: reuse happens at 1,648-token block granularity
|
||
(the attention block size is padded to the mamba page size) — an identical
|
||
13,021-token prompt went 2.30 s cold to 0.68 s warm with 9,888 tokens (76 %)
|
||
reused, while a 2,521-token prompt registered zero hits (under two blocks).
|
||
- **Network decision.** `eno2` (5GbE, 192.168.0.247) kept, `eno1` pinned primary
|
||
via metric 100 vs 200 in `scripts/dankserver-route-primary.sh`; applied with
|
||
`nmcli device reapply` (there is no `nmcli connection reapply` in NM 1.54).
|
||
- **Docs.** [[frank-medium-radiance-mxfp4]] is the ACTIVE runbook (deployed flags,
|
||
battery, capabilities, rollback, with the replica-era sections labelled);
|
||
[[frank-medium-llamacpp-dflash2]] carries the SUPERSEDED banner and past-tense
|
||
lead; [[Dual-R9700-Cooperation]] now states the measured verdict instead of the
|
||
open question; [[Observability]] records the namespace split and the
|
||
netavark/DNAT lesson; `AGENTS.md` topology, frank-medium section, Model Config,
|
||
Benchmarking Policy (frank-medium is a **user** Quadlet now — `systemctl --user
|
||
stop/mask`) and the runbook index all flipped.
|
||
- **Doc-hygiene incident worth repeating.** Two parallel doc agents left the same
|
||
three wounds twice: prose prepended **above** the `---` frontmatter opener,
|
||
whole body sections duplicated wholesale, and a literal
|
||
`"<|channel>}<tool_call|>…"` token pasted into a runbook. `check-vault-links.py`
|
||
catches only unresolved links, not structure — so the fix was `git checkout
|
||
HEAD --` the destroyed index and re-apply the flips by hand, with the agent
|
||
versions moved to `.AGENTTRASH/20260920-index-dupes/`. After any agent pass over
|
||
docs: re-check frontmatter is first, headings unique, and links resolve.
|
||
|
||
## 2026-09-20 — dankserver board swap: the PCIe-atomics wall is gone
|
||
- **Hardware.** dankserver now runs an **ASUS ProArt B850-Creator WiFi NEO**
|
||
(was TUF GAMING B650-E WIFI), same Ryzen 7 7700X and same 2x R9700. The board
|
||
bifurcates the CPU GPP ports **x8/x8 at Gen5**, so both cards sit on CPU root
|
||
ports: card 0 `00:01.1 → 01:00.0 → 02:00.0 → 03:00.0`, card 1
|
||
`00:01.3 → 05:00.0 → 06:00.0 → 07:00.0`. Every bridge reads
|
||
`AtomicOpsCap: Routing+` and **both endpoints read `AtomicOpsCtl: ReqEn+`**
|
||
(`CAP_EXP+28.w = 0x1440`) — the exact precondition
|
||
[[Dual-R9700-Cooperation]]'s test plan named before any model would be
|
||
touched. Bandwidth per card ~31.5 GB/s (kernel: `252.056 Gb/s … limited by
|
||
32.0 GT/s PCIe x8`): card 0 unchanged from Gen4 x16, card 1 up ~16x from
|
||
Gen4 x1.
|
||
- **Verdict, stated honestly.** vLLM TP=2 / PP=2 and llama.cpp `--split-mode
|
||
row` are **unblocked, not proven** — no collective has been dispatched since
|
||
the swap, `GGML_HIP_RCCL` is still `OFF` in `scripts/build-llama-rdna4.sh`,
|
||
and the 272 t/s "tensor" number on record measured a butterfly fallback, not
|
||
an allreduce. `librccl.so.1` + `hipcc` already exist in the bundled TheRock
|
||
toolchain, so an RCCL-backed rebuild needs no new host packages. The probe
|
||
still owes a stop-and-mask window.
|
||
- **Side effects of the swap, recorded.** GPU BDFs moved (`0b:00.0` → `07:00.0`),
|
||
iGPU `10:00.0` → `11:00.0`, WiFi interface `wlp8s0` → `wlp14s0`, kernel now
|
||
7.0.0-31-generic, SMT back **on** (16 logical CPUs; llama.cpp still `--threads
|
||
8`), and the board's second 5GbE port is cabled as `eno2` = 192.168.0.247 with
|
||
the lower-metric default route while `eno1` keeps the fleet's static
|
||
192.168.0.235. Two addresses, two default routes — needs a decision.
|
||
- **Docs cut over** (no service change; frank-medium stayed up and healthy
|
||
throughout): [[Dual-R9700-Cooperation]] short answer rewritten plus a
|
||
re-measured 2026-09-20 topology table; [[Dankserver System Specs]] gained a
|
||
Motherboard section and corrected GPU/iGPU/network/kernel rows;
|
||
[[frank-medium-llamacpp-dflash2]] hardware row and rationale now split past
|
||
from present; `AGENTS.md` topology table, the frank-medium section, and the
|
||
PCIe lesson bullets no longer assert a wall that no longer exists. The
|
||
dated work-log sections that describe the old board stay as history.
|
||
|
||
## 2026-09-19 — fleet cutover: frank-large CIRU Orca, frank-medium orcarouter Q8_0, frank-small Gemma4 revert
|
||
- **frank-large (Dankclaw :7770): halogen → CIRU v4.4.1 Orca.** The private
|
||
CIRU v4.4.1 runtime (gfx1151 HIP build + QSA window hotfix, 381/381 ninja
|
||
targets) now serves jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b (79.4 GiB main +
|
||
4.1 GiB MTP + 52.4 GiB PLE + f16 mmproj, all sha256-verified) as a ROOT systemd
|
||
unit — one 262,144-token conversation (1 slot, the MTP constraint), MTP depth 4
|
||
(acceptance 0.656), 4096 MiB PLE cache, text only. The launcher's runtime-root
|
||
gate is fed from the SDK venv by the serve script. Halogen is the rollback
|
||
target. (`b6ce3af`, `f68bda5`; runbook [[frank-large-ciru-orca]]).
|
||
- **frank-medium (Dankserver :7770): ajgazin Swift UD-Q8_K_XL → orcarouter Q8_0.**
|
||
Model swap to orcarouter/Qwen3.8-27B-Uncensored-GGUF Q8_0 with the z-lab
|
||
DFlash2 Q4_K_M drafter (the target's own MTP head stays in-file, one SPEC_TYPE
|
||
away); split re-balanced 35:30 → 32:33 (operator call: the most even split 65
|
||
layers allow); pool 393216 → 327680 (2 slots of 163840, inside the trained
|
||
262144 window — no rope extrapolation); f16 KV + 16 GiB host-RAM prompt cache.
|
||
(`d374ab9`).
|
||
- **frank-small (Dankclaw :7771): vLLM → llama.cpp Gemma4-12B Q4_K_M.** The
|
||
same-day vLLM detour (google/gemma-4-12B-it-qat-w4a16-ct, w4a16 Marlin) was
|
||
reverted back to mainline llama.cpp serving
|
||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M with the MTP
|
||
assistant sidecar + ngram-mod, q8_0 KV, 4 slots of 262144. The vLLM quadlet is
|
||
retired. (`374041c`; runbook [[frank-small-gemma4-12b]]).
|
||
- **3090 RGB off.** The EVGA FTW3 Ultra V2's iCX3 controller (i2c-1, addr 0x2D)
|
||
was set to MODE_OFF and SAVED to EEPROM — persists across reboots with no
|
||
Linux-side daemon; works headless.
|
||
- **Docs wrap-up** (`764cf41`, `88331e2`): AGENTS.md (frank-large + frank-medium
|
||
sections, topology rows), new [[frank-large-ciru-orca]] runbook, frank-medium
|
||
runbook (header + deployed-flags table), both indexes, halogen runbook
|
||
retirement banner. 331 bats + 32 e2e green.
|
||
|
||
## 2026-09-16 (late) — frank-medium: ubatch's optimum is 512, and it moved the split to 35:30
|
||
- **New harness, shipped before use** (`scripts/bench-batch-ubatch.sh`, `87845fc`,
|
||
fixed in `77a07ff`): one `llama-bench` invocation per cell — a test that cannot
|
||
allocate calls `exit(1)`, so one infeasible ubatch would take the whole sweep
|
||
with it — a VRAM sampler per cell, and the service stopped for the run and
|
||
started again from a trap. It must run from a transient unit: the deploy's
|
||
`LimitMEMLOCK=infinity` is load-bearing for ROCm staging buffers and an SSH
|
||
shell on these hosts only gets 8 MiB.
|
||
- **The sweep** (23 of 28 cells, a 64k prefill measured *at* 64k depth, 3 reps):
|
||
prefill **peaks at ubatch 512**. 256 and 384 lose ~11 % (SMs not saturated),
|
||
2048 loses 10 %, and 768–1792 sit in a shallow basin 3–5 % below the peak. The
|
||
curve is deterministic rather than noisy — 512 measured 1192.88 / 1193.23 /
|
||
1193.28 / 1193.88 across batch 2048 / 4096 / 6144 / 8192, and ubatch 2048
|
||
measured 1077.38 / 1077.54 / 1077.59 — which is what makes the 2.4 % gap
|
||
between 512 and the deployed 1536 real.
|
||
- **The logical `--batch-size` is irrelevant**: 2048 through 8192 land within 2 %
|
||
of each other at a fixed ubatch and use identical VRAM, because the compute
|
||
buffers are sized by ubatch alone. Decode never moves at all — 15.88 tok/s at
|
||
64k depth on every cell, 17.80 at depth 0 — while depth itself costs a uniform
|
||
~32 % of prefill.
|
||
- **Deployed change** (`a11fae0`): `UBATCH_SIZE` 1536 → 512, which is both faster
|
||
and 3.71 GiB cheaper, and `TENSOR_SPLIT` 36,29 → 35,30, because the ubatch
|
||
buffers cost ROCm1 nearly twice what they cost ROCm0 (1.30 against 2.41 GiB
|
||
for the 1024-unit move, measured at 262144 over two slots) and shrinking them
|
||
handed ROCm1 1.31 GiB more than ROCm0. Live after the deploy: **6.44 / 6.90
|
||
GiB free** (was 4.71 / 4.91), 6.23 / 6.64 under two concurrent requests, and a
|
||
0.46 GiB gap — one DeltaNet layer, which is the granularity floor here.
|
||
- The balance seam in `tests/frank_medium_unit.bats` was re-fitted at the new
|
||
deployed point with both ubatch coefficients measured at this shape; every
|
||
neighbouring split still fails it (36,29 → 1.85 GiB, 34,31 → 0.94, 37,28 →
|
||
3.24, 1,1 → 3.03, 39,26 → 6.03). The unit test's literal `TENSOR_SPLIT=36,29`
|
||
assertion became a *shape* assertion — two shares summing to the model's 65
|
||
blocks with ROCm1 taking the smaller — so re-measuring the balance no longer
|
||
breaks a test about parallelism.
|
||
- **Measurement trap worth keeping**: re-loading immediately after a kill fails
|
||
with `unable to allocate ROCm0 buffer`. amdgpu does not hand ~26 GiB back at
|
||
once, and a 180 s settle is not enough; it reads exactly like a configuration
|
||
that does not fit. The measurement helper grew a wait-for-empty gate.
|
||
- Docs: [[frank-medium-batch-ubatch]] (new page), the runbook's deployed flags
|
||
and split history, and AGENTS.md's batch lesson. [[Dual-R9700-Cooperation]]
|
||
gained the live `AtomicOpsCap` / `AtomicOpsCtl` chain — `ReqEn+` on card 0
|
||
against `ReqEn−` on card 1, because the kernel refuses the requester bit when
|
||
a bridge cannot route atomics — and its M.2_2 riser premise is now labelled a
|
||
vendor claim rather than a fact.
|
||
|
||
## 2026-09-15 — frank-medium: Dankserver GPU swap (2x R9700), frank-small retired
|
||
- The RTX 3090 left Dankserver; two AMD Radeon AI PRO R9700 cards (Navi 48,
|
||
gfx1201, 32,624 MiB each) replaced it. FreeToken/CUDA cannot run there, so
|
||
`frank-small` is retired and `frank-medium` took the :7770 endpoint.
|
||
- Runtime: charlie12345/ROCmFPX llama.cpp `c49ebdb` built for **gfx1201** with
|
||
the fork's self-contained ROCm 7.14 toolchain (TheRock nightly, RPATH-bundled
|
||
libs) — the host's broken mixed ROCm (AMD-repo 6.3.3 hijacking libhsa via
|
||
`/etc/ld.so.conf.d/10-rocm-opencl.conf`) is fully bypassed. Model:
|
||
kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF (ftype 115, 26.28 GiB) +
|
||
Q4_0 MTP draft head; mmproj unused (vision stays on frank-large).
|
||
- **Parallelism decision, measured not argued:** llama-bench with the correct
|
||
multi-device syntax (`-dev ROCm0/ROCm1` — comma form silently benches each
|
||
card separately): layer 1991/18.8 t/s (pp4096/tg128) vs tensor 271.8/26.3 vs
|
||
single card 1160.9/19.4. Tensor's decode win costs a 7x prefill collapse
|
||
(no RCCL → "meta-backend butterfly" allreduce); served with MTP, layer also
|
||
wins decode (35.7–46.2 tok/s vs 28.7). Operator picked layer ("pipeline
|
||
parallelism"); the data agrees on both axes. Draft head pinned to ROCm1
|
||
(card holding the last target layers); under tensor mode an unpinned draft
|
||
aborts the KV build (GGML_ASSERT(buffer) in ggml-backend.cpp).
|
||
- Verified live before and after the systemd cutover: health ~18 s, MTP
|
||
acceptance 60–69 %, cold 949-token prefill in 2036 ms, and **prefix caching
|
||
works with the draft head** (turn 2 reprocessed 22 tokens, cache_n 945) —
|
||
the model card's mtp-prompt-cache-fix.patch defect is already fixed in
|
||
`c49ebdb` (`common_prompt_checkpoint::data_spec`).
|
||
- Deploy: `deploy.sh dankserver --service frank-medium` (new
|
||
`retire_root_llm_small()` archived the CUDA-era unit to
|
||
`~/.AGENTTRASH/20260915-frank-small-retire/host-units/`); e2e gate 30/30
|
||
including 11-admin (the blankpad→192.168.0.112 path issue has cleared).
|
||
- Host cleanup (the "Both" half of the plan): purged 167 stale AMD-repo
|
||
`~24.04` ROCm 6.3.3 packages, removed the radeon apt pin and the
|
||
ld.so.conf hijack (staged in `~/.AGENTTRASH/20260915-rocm633-cleanup/`);
|
||
`libhsa-runtime64` now resolves only to Ubuntu's ROCm 7.1 and `rocm-smi`
|
||
7.1.1 (Ubuntu package) works. frank-medium unaffected (self-contained).
|
||
- Client cutover: blankpad `models.yml`/`config.yml` renamed the model id to
|
||
`frank-medium` (ctx 131072, maxTokens 32768); Hermes live config on Dankclaw
|
||
repointed `dankserver-small` routes to `frank-medium`.
|
||
- New: `/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md` runbook;
|
||
updated AGENTS.md, README, docs indexes, Dankserver System Specs,
|
||
deployment/Hermes/OpenWebUI/Plex/Stack pages; freetoken-frank-small and
|
||
kingjones-q8agent marked retired/superseded.
|
||
|
||
## 2026-09-12 — Marreta content-cleaning proxy deployed on Dankclaw
|
||
- Added `manualdousuario/marreta` (Laravel / `serversideup/php:8.4-fpm-nginx`)
|
||
as a rootless Quadlet pair: `marreta` + `marreta-browser` (Lightpanda),
|
||
both joining **gluetun's netns** so all fetch egress exits the same
|
||
ProtonVPN tunnel as [[SearXNG]].
|
||
- **Port-collision fix:** the shared netns already has SearXNG on 8080, so
|
||
marreta is relocated to `NGINX_HTTP_PORT=8085` (verified against the
|
||
serversideup source: the image does not bake `conf.d/*.conf`; the
|
||
entrypoint `envsubst`s `${NGINX_HTTP_PORT}` at start). Gluetun publishes
|
||
`8082:8085` on marreta's behalf; Caddy dials `192.168.0.143:8082`.
|
||
- **Egress rationale:** netns sharing, not `PROXY_URL` — the browser's CDP
|
||
fetch passes no proxy, so only netns keeps the browser off the home IP.
|
||
- **Guard chain:** `marreta.franksdank.cloud` → Anubis `:8923` → Authentik
|
||
→ post-auth `:8017` `@marreta` → `192.168.0.143:8082`. Access model is
|
||
**any authenticated user** (no group gate). Requires a one-time
|
||
`ProxyProvider` (external_host `https://marreta.franksdank.cloud/`,
|
||
internal_host `http://caddy:8017`) on the embedded outpost.
|
||
- Secrets seeded to `/home/dank/Containers/marreta/env` (0600, dank-owned)
|
||
from `MARRETA_APP_KEY` / `MARRETA_ADMIN_EMAIL` / `MARRETA_ADMIN_PASSWORD`.
|
||
- New test seam `tests/marreta_quadlet.bats` (17 contracts). deploy-plan.py:
|
||
marreta joins the gluetun netns family. Doc: [[Marreta]].
|
||
- **Deployed live 2026-09-12 (rev `91e9410`)** after standing up passwordless
|
||
SSH from blankpad to the fleet. Ran `deploy.sh dankclaw --service gluetun
|
||
--service marreta --service caddy` (gluetun first: the new `8082:8085`
|
||
publish needs a tunnel recreate, which also cycles SearXNG). Authentik
|
||
`ProxyProvider` `marreta` registered (pk 13). Verified: unauth → Anubis
|
||
challenge; post-auth `:8017` → 200; container healthy; netns sharing proven;
|
||
SearXNG unaffected.
|
||
- **Gotcha fixed in-flight:** the first seed used a bare-base64 `APP_KEY`,
|
||
which 500s every Laravel app route (`Unsupported cipher or incorrect key
|
||
length`). Re-keyed with the required `base64:` prefix before any encrypted
|
||
data persisted. See [[Marreta]] § Secrets.
|
||
- **Login-wall fixed 2026-09-12 (post-deploy).** The homepage Marreta link
|
||
parked on `marreta.franksdank.cloud/if/flow/default-authentication-flow/`
|
||
instead of the app. Root cause chain, in the order found:
|
||
1. `marreta-proxy` had `invalidation_flow = null`, which crashed the Rust
|
||
embedded outpost's serde (`invalid type: null, expected a formatted UUID
|
||
string`) and took down **every** provider on the outpost. Set to
|
||
`default-provider-invalidation-flow`.
|
||
2. `authentication_flow` / `authorization_flow` were null; set to
|
||
`default-authentication-flow` / `default-provider-authorization-implicit-consent`.
|
||
3. **The real blocker:** the outpost's config API filters
|
||
`ProxyProvider.objects.filter(application__isnull=False)`. `marreta-proxy`
|
||
had no linked **Application**, so the outpost never loaded it and the
|
||
request fell through to raw flow serving on the marreta host — where the
|
||
IdP session cookie (scoped to `auth.franksdank.cloud`) is invisible.
|
||
Created `Application(name="Marreta", slug="marreta", provider=marreta-proxy)`.
|
||
- After `systemctl --user restart authentik-server authentik-outpost`, the
|
||
outpost loaded `marreta-proxy` and the OAuth dance completes: `/` →
|
||
`/outpost.goauthentik.io/start` → `auth…/o/authorize` → callback → Marreta.
|
||
Regression test: `tests/e2e/specs/16-marreta.spec.ts`. Providers remain
|
||
**live-only (no repo blueprint)** — a provenance gap to close.
|
||
|
||
## 2026-09-04 — *arr fleet consolidated on FrankBook-Pro; all egress VPN-gated
|
||
- Operator decision: **FrankBook-Pro is the sole *arr deployment.** The
|
||
Dankserver compose copy (booted earlier the same day) was removed;
|
||
`/opt` config roots trashed on the host; compose/services pruned.
|
||
- All six fleet containers (qB + five arrs) now share gluetun's network
|
||
namespace — **all egress exits the ProtonVPN tunnel by construction**
|
||
(AllowedIPs=0.0.0.0/0). The arrs publish no ports of their own; their UI
|
||
ports (9696/8989/7878/8686/8787) moved onto the gluetun unit. Proven live:
|
||
residential `71.227.227.172` vs netns-wide `51.179.177.14`.
|
||
- New test seam `tests/frankbook_quadlet.bats`: every arr+qB unit must join
|
||
gluetun's netns and publish no port; gluetun must publish the whole
|
||
fleet's port set. deploy-plan.py: gluetun changes carry the whole netns
|
||
family.
|
||
- arr→qB download clients re-pointed `10.88.0.1:8080` (old bridge-gateway
|
||
hack, dead inside the tunnel) → `127.0.0.1:8080`; verified by real qB
|
||
login from inside the netns. Caddy vhosts + homepage probes repoint to
|
||
192.168.0.112; homepage all green.
|
||
- Defects discovered: Prowlarr's four Application rows are GONE (empty
|
||
table; sync broken until rebuilt with loopback baseUrls) and readarr's
|
||
qB proxy auth defect needs a retest post-move. EZTV/1337x CF-block
|
||
parking may be stale now that egress is VPN — retest before enabling.
|
||
- E2E specs (11-admin/12-crawl) de-hardcoded: expected open-link counts now
|
||
derive from the live services.json registry. AGENTS.md: deployments are
|
||
complete only when unit tests AND Playwright E2E both pass.
|
||
|
||
## 2026-09-04 — dankserver *arr stack reactivated; rootless identity model documented
|
||
- ARR-Stack: the Dankserver compose deployment (committed 2026-08-29, never
|
||
booted — root-owned `/opt/<svc>/config` dirs crashed every app at logger
|
||
init before any port bound) is live as of 2026-09-04 behind six own-auth
|
||
TLS vhosts (`<svc>.franksdank.cloud`), all verified 200, homepage probes
|
||
green. Documented the verified rootless identity model — container root
|
||
(PUID=0) maps to host uid 1000 (dank) by default on podman 5.7.0
|
||
(`/proc/<pid>/uid_map` = `0 1000 1`, `1 100000 65536`), so data roots are
|
||
`1000:1000` and **`userns_mode` must never be added to compose services**
|
||
(podman-compose 1.5.0 runs services in pods; `--userns` + `--pod` is
|
||
rejected). FrankBook-Pro differences noted: no gluetun egress gate on the
|
||
Dankserver qB, empty (fresh) app configs, Sonarr in the homepage user
|
||
section and the rest in admin.
|
||
## 2026-09-03 — full fact-check sweep of the docs tree (docs-only, deploy no-op)
|
||
- Scope: every page under `/serverAppConfigs-docs/` re-verified against live
|
||
hosts (read-only SSH), quadlet unit files, and the Caddyfile; surgical edits
|
||
only — no pages created, renamed, or deleted. Per the deploy planner, a
|
||
docs-only change is a deploy no-op.
|
||
- Dankserver core rewritten to live quadlet facts: the *arr stack + qBittorrent
|
||
+ gluetun run on **FrankBook-Pro** (units installed to
|
||
`/home/core/.config/containers/systemd/` via the frankbook-pro deploy
|
||
handler; `PUID=0/PGID=0`, data roots `/Volumes/COMPLETE/<svc>-data:/config`;
|
||
qB pinned `4.6.7` sharing gluetun's netns with 8080 + 6881 published on the
|
||
gluetun unit; readarr pinned `amd64-1.0.980`; no vhosts — direct IP:port,
|
||
exempt by construction from the guard chain). Jellyfin, Plex, RomM (8085→8080
|
||
guarded), Calibre-Web (`0.6.27-ls397` + universal-calibre mod; no standalone
|
||
calibre-server), mariadb/redis/Tautulli/Entropy verified against live units.
|
||
- Caddy guard chain documented to the live mechanism end-to-end: Anubis :8923 →
|
||
internal :8016 → post-auth :8017 routing; guarded vhosts (search, grafana,
|
||
prometheus, loki, seerr) route post-auth to container-name upstreams;
|
||
exemptions named by construction reason (auth/IdP itself, gitea smart-HTTP,
|
||
ai→:3005 own auth, hermes/hermesadmin internal-only via DNS absence, pihole
|
||
basicauth edge → :8083).
|
||
- `Gitea.md`: data roots + secrets corrected to the live unit paths
|
||
(`/home/dank/Containers/gitea/{data,postgres,env}`); UID gotcha corrected —
|
||
the units carry no `User=`/UserNS override, so container uid 1000 is literally
|
||
host uid 1000 (no subuid offset for these containers).
|
||
- Dedupe: `Prowlarr.md`/`prowlarr.md` collapsed into one canonical lowercase
|
||
page via rename (git history retains both versions; index link unchanged).
|
||
- LLM docs: archive pages (`DFlash DSpark Speculative Decoding`,
|
||
`ling-3.0-flash-trial`) reworded from current-tense to dated historical
|
||
framing with pointers to the active runbook [[kingjones-strix-rocmfp4]];
|
||
LLM `index.md` + `deployment.md` fleet tables and health-gate text updated to
|
||
the 2026-08-30 fleet state (STRIX on the pinned ROCmFPX fork; Huihui
|
||
abliterated FP8 on FreeToken; sightsound = FLM v1.0.1 on XDNA2, port 7771,
|
||
health `/v1/version` — FLM has no `/health` route).
|
||
- Suite green after the sweep: `./tests/run.sh` → 233 pass / 0 fail, including
|
||
the docs-completeness seam that requires every page to stay index-linked.
|
||
## 2026-09-01 (late) — FrankBook-Pro stack completed: tunnel live, qB wired into the arrs
|
||
- WireGuard credentials provisioned (second Proton config, server `PA-12`,
|
||
endpoint `51.179.177.2:51820`) → `/home/core/Containers/gluetun/env`
|
||
(0600, VM-local); `deploy.sh frankbook-pro --service gluetun --service
|
||
qbittorrent` brought both up; egress verified (Proton PA exit, ~19 MB/s
|
||
on a live torrent). The `.conf` stays in `~/Downloads`; the private key
|
||
never enters the repo.
|
||
- gvproxy binds wildcard `*:6881/udp` (Go sets `SO_REUSEADDR` on TCP but not
|
||
UDP) → the socat 6881 tcp+udp LaunchAgents collided at every boot and
|
||
were retired; swarm inbound (if ever needed) is the Proton NAT-PMP path.
|
||
macOS application firewall still filters gvproxy inbound from non-local
|
||
sources (stealth mode) — cross-host 8080 needs `socketfilterfw` allow.
|
||
- qB wiring: first-seed bug (hash under legacy `WebUI\Password` key, ignored
|
||
by qB ≥ 4.2) fixed live (`WebUI\Password_PBKDF2`) and in
|
||
`scripts/deploy-remote.sh`; `DefaultSavePath` repointed to
|
||
`/Volumes/COMPLETE/Finished Torrents` — qB 5.2 API adds with a category
|
||
do not inherit the category save path (no auto-TMM); download clients
|
||
created in all four arrs (sonarr/radarr/lidarr via API, tests pass;
|
||
readarr by direct `readarr.db` insert — its EOL validator rejects qB 5's
|
||
`204` login as auth failure); Prowlarr Applications sync sonarr/radarr/
|
||
lidarr (readarr omitted: prowlarr's indexer push 500s on EOL schema).
|
||
Legacy `Prowlarr-*` indexer rows (hostname `prowlarr:9696`, unresolvable
|
||
on the Mac default network) deleted. E2E: Sintel fetched + downloaded
|
||
through the tunnel at ~19 MB/s, completed into `Finished Torrents`.
|
||
- Docs: [[FrankBook-Pro]] rewritten to the active-tunnel reality.
|
||
|
||
|
||
## 2026-09-01 - WG endpoint identified as ProtonVPN Seattle; Mac gate is a second dashboard config
|
||
- RDAP: `149.102.254.77` = CDNEXT-CG-SEA (Datacamp Limited, Proton's Seattle
|
||
hosting partner); Dankclaw's tunnel v6 `2a07:b944::2:2` sits in
|
||
CH-PROTONVPN-20160719; observed exit IP `149.22.88.19` = CDNEXT-SEA-CG.
|
||
The "custom WG server" is a ProtonVPN Seattle WireGuard server - there is
|
||
no peer-registration step on any box of ours (full 65535-port scan: no
|
||
SSH anywhere, just silent TLS listeners - 443, 4443-4468, 5995, 7770,
|
||
8443 - Proton's protocol-stealth port map, no admin surface).
|
||
- Mac gate corrected: provision a **second** Proton WG config (dashboard,
|
||
named e.g. `frankbook`) - one config = one keypair; two hosts on
|
||
Dankclaw's config bounce the server-side endpoint and kill SearXNG. Values
|
||
paste straight into the gluetun `custom` env. Earlier prepared keypair on
|
||
dankclaw (`~/frankbook-wg/`) retired to `~/.AGENTTRASH/` (Proton generates
|
||
the keypair per config; no BYO pubkey).
|
||
- [[FrankBook-Pro]] provisioning section rewritten.
|
||
|
||
## 2026-09-01 — Jellyfin authentication moved to Authentik LDAP
|
||
- `/dankclaw/authentik/quadlet/authentik-outpost-ldap.container` (`4ad85fa`,
|
||
digest-pinned in `f4ef2b8`) — new `authentik-outpost-ldap.service` on
|
||
Dankclaw serving LDAP :389/:636 for Jellyfin. Image
|
||
`ghcr.io/goauthentik/ldap` pinned by **amd64 single-platform digest**:
|
||
podman 5.7.0 refuses the tag because it resolves to a nested manifest
|
||
list ("unexpectedly received a manifest list instead of a manifest for a
|
||
single image") — resolve via `podman manifest inspect | jq` and pin.
|
||
- Authentik objects provisioned via `ak shell` ORM (live, idempotent
|
||
get_or_create): LDAP provider `Jellyfin LDAP`, service account
|
||
`svc-jellyfin-ldap` + bind token, groups `jellyfin_users` /
|
||
`jellyfin_admins`, application `jellyfin`.
|
||
- `/scripts/deploy-remote.sh` (`691da47`) — `authentik-outpost-ldap` added
|
||
to `service_is_allowed` (the deploy runner has its own allowlist beyond
|
||
deploy-plan.py's).
|
||
- Two authentik-side rules proven by live bind tests (`ldapsearch` through
|
||
the published port):
|
||
1. Bind token **intent must be `verification`** — `app_password` tokens
|
||
are rejected by the authorization flow's password stage with
|
||
`Invalid password`.
|
||
2. Search requires the **`search_full_directory`** object permission on
|
||
the provider (granted via RBAC role `svc-jellyfin-ldap-search`);
|
||
without it the outpost returns only the bind user, so `memberOf`
|
||
filters match zero entries.
|
||
- Jellyfin side (host-only, no repo change): LDAP-Auth **23.0.0.0**
|
||
installed from the GitHub releases zip (the `repo.jellyfin.org` stable
|
||
manifest path 404s), config XML at
|
||
`config/plugins/configurations/LDAP-Auth.xml`, username attribute `cn`
|
||
(authentik 2026.x maps `uid` to an opaque hash), and — the big one —
|
||
**`LdapSearchAttributes` is comma-separated** (`Split(',')` in the plugin
|
||
source; semicolons silently produce "Found no users matching X").
|
||
End-to-end verified with a throwaway `jellyfin_users` member
|
||
(`AuthenticateByName` 200, non-admin policy), then removed from both
|
||
authentik and the Jellyfin DB directly (its admin-token cleanup 401'd —
|
||
the admin password in the session archive was OCR-corrupted; verified
|
||
offline against the stored PBKDF2-SHA512 hash that the real bootstrap
|
||
value still authenticates, unchanged).
|
||
- Docs: [[Authentik]] gained the LDAP-outpost section (objects table +
|
||
rules), [[Jellyfin]] gained the LDAP-Auth plugin section. Docs-only
|
||
commit → deploy no-op.
|
||
|
||
## 2026-09-01 — frank-small: two compressed-tensors FP8 swaps fail to load; Huihui restored
|
||
- `/scripts/frank-small.env`, `/scripts/serve-frank-small.sh`,
|
||
`/scripts/frank-small.service` (`af6cd72` → `34ab6b4` → reverted
|
||
in `4d5bea4`) — attempted model swaps on FreeToken v0.1.2, both
|
||
live-verified as loader failures and reverted the same day to the Huihui
|
||
FP8 checkpoint:
|
||
- `ornith-ai/Ornith-1.5-35B-A3B-FP8` (compressed-tensors per-channel):
|
||
`ValueError: Missing MoE expert source layers: {'gate_up': ...}` — the
|
||
checkpoint's expert tensors do not follow the
|
||
`mlp.experts.{N}.gate_proj/up_proj/down_proj` naming the MoE provider
|
||
scans for.
|
||
- `lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8`
|
||
(compressed-tensors block-fp8 128×128, standard naming): experts
|
||
resolve but `_dequant_fp8_weight` multiplies `weight × weight_scale`
|
||
directly — block scales do not broadcast →
|
||
`RuntimeError: The size of tensor a (2048) must match ...`.
|
||
- Both failed cleanly at load (37 GB fits RAM — no swap-thrash, no
|
||
outage); service was back on Huihui within the hour and re-verified with
|
||
a real completion (reasoning parses; `17*23 → 391`, natural stop).
|
||
- `/serverAppConfigs-docs/LLM/freetoken-frank-small.md` — quant
|
||
matrix updated: modelopt block-fp8 is the ONLY FP8 path that loads; both
|
||
compressed-tensors shapes documented with their exact errors.
|
||
- **Deploy health-check trap (second occurrence)**: `deploy.sh` reported
|
||
"healthy after approximately 6s" for BOTH failed loads — the health gate
|
||
passes while the doomed worker is still starting. A model swap is only
|
||
proven by the journal (Uvicorn startup) + a real completion.
|
||
- Staged for trash (awaiting purge approval): `/home/dank/models/ornith-1.5-35b-a3b-fp8`
|
||
(37 GB) and `/home/dank/models/hauhaucs-qwen36-35b-a3b-aggressive-fp8`
|
||
(35 GB).
|
||
|
||
## 2026-09-01 — dankserver media stack standardized to quadlet; dankclaw data roots migrated
|
||
- **Incident context**: Dankserver hard-crashed and rebooted 2026-09-01; every
|
||
bare compose-era media container (RomM trio, Jellyfin, Calibre-Web,
|
||
Copyparty) stayed dead at boot — the failure mode the Boot Persistence rule
|
||
exists for. Recovery doubled as the Phase 1 cutover.
|
||
- **Phase 1 (dankserver)**: six quadlet units shipped
|
||
(`dankserver/{romm,jellyfin,calibre-web,copyparty}/quadlet/`) + first-cutover
|
||
deploy handlers (retire bare containers → rescue RomM's anonymous `/romm`
|
||
volume → merge-move `/opt/<svc>` into `/home/dank/Containers/<svc>` →
|
||
ordered start with health gates). All six verified: units `active`,
|
||
`is-enabled=generated` (wants in the generator dir), ports 8085→8080,
|
||
8096, 8083, 3923 answering, jellyfin holds `/dev/dri` render nodes,
|
||
entropy binds provably ro except copyparty's `/w`. MariaDB continuity
|
||
proven via the real `dank` admin row (created 2026-08-31), not a fresh
|
||
bootstrap. Stale compose-era `container-*.service` generate-systemd units
|
||
retired to `~/.AGENTTRASH/20260901-stale-generate-systemd`.
|
||
- **Deployer findings fixed** (each caught live, each with a new seam):
|
||
v1 generator rejects `Command=` (→ `Exec=`), `Device=` (→ `AddDevice=`,
|
||
whole-unit rejection is silent), and splits spaces in `HealthCmd=`
|
||
(→ omitted for copyparty); `migrate_data_root` must merge into pre-seeded
|
||
destinations and rename mapped-subuid roots (100998) with `sudo mv`
|
||
(numeric-preserving, `keep` mode skips the chown — chowning mapped data
|
||
breaks the in-container users); quadlet unit dir must be created on a host
|
||
that never had quadlet; missing `PublishPort` = app answers inside the
|
||
netns only (copyparty 3923); seeded env files must carry **compose-provided
|
||
vars only** — the image-baked `HOME=/root` in a `[Service]
|
||
EnvironmentFile=` kills the podman client itself.
|
||
- **Phase 2 (dankclaw)**: data roots of gitea, authentik, hermes trios moved
|
||
to `/home/dank/Containers/<svc>/` while running (rename is inode-bound —
|
||
zero downtime), units repointed, redeployed through `deploy.sh --service`;
|
||
auth chain verified end-to-end (outpost ping → 302 to flow).
|
||
`deploy_hermes` ordering bug fixed: generated units must be reloaded
|
||
**after** unit files are copied, not before (stale ExecStart crash-looped
|
||
the trio on the old path).
|
||
- **Docs**: [[RomM]], [[Jellyfin]], [[Calibre-Web]], [[Copyparty]] updated to
|
||
the quadlet reality; [[Storage Policy]] now documents the Containers layout
|
||
+ the two ownership regimes; [[Deployment Flow]] service-manager table
|
||
corrected (dankserver media is no longer "bare containers, inventory only").
|
||
|
||
## 2026-08-30 (night II) — frank-npu deployed; NPU SVA/IOMMU root cause
|
||
- **Deployed** `frank-npu` on Dankclaw: FLM v1.0.1 (`/usr/bin/flm`,
|
||
dpkg `fastflowlm`) serving the whole local registry (36 tags) on port 52626;
|
||
primary model Gemma4 E4B IT NPU2 (vision + audio towers). First deploy
|
||
crash-looped every 10 s: `Open /dev/accel/accel0 failed (err=-19)`.
|
||
- **Root cause** (live-verified): the XDNA2 driver's SVA bind fails with ret -19
|
||
whenever the AMD IOMMU is off; the kernel cmdline carried `iommu=off` (a
|
||
perf/DMA tuning experiment from 08-20, applied at the 08-26 reboot). No IOMMU
|
||
groups ⇒ no PASID state ⇒ accel0 open returns ENODEV for every client. The
|
||
same root cause explains the `amdgpu ... no vmid pasid mapping supported`
|
||
kernel line — a consequence, not a GPU fault.
|
||
- **Fix**: `iommu=pt` in `GRUB_CMDLINE_LINUX_DEFAULT` + reboot (passthrough keeps
|
||
the iGPU default domain identity-mapped — zero DMA-translation overhead for
|
||
frank-large — while enabling IOMMU hardware + PASID, which SVA needs;
|
||
PASID domains are paged regardless of the default-domain type). Post-boot:
|
||
30 IOMMU groups, zero SVA-bind errors since boot, unit `enabled`,
|
||
`NRestarts=0` since boot.
|
||
- **Live-verified behavior**: whole-registry serving with per-request routing by
|
||
exact FLM tag (unknown tags silently serve `llama3.2:1b`); vision tower proved
|
||
end-to-end (generated red PNG as `image_url` data-URI → correct color answer,
|
||
279 prompt tokens consumed); audio tower proven with a control pair (440 Hz
|
||
tone via `input_audio` = 37 prompt tokens vs 15 for the identical text-only
|
||
control). Health probe is `/v1/version` — FLM has no `/health` route.
|
||
- `/scripts/frank-npu.env`, `/scripts/serve-frank-npu.sh`,
|
||
`/scripts/frank-npu.service` + deployer allowlist entries
|
||
(`c3ed3a6`, `cf0c541`) — service files and deploy wiring; unit is boot-persistent
|
||
(`WantedBy=multi-user.target`, enabled, `Restart=on-failure`, `RestartSec=10`,
|
||
`LimitMEMLOCK=infinity` — FLM requires unlimited memlock).
|
||
- **Docs**: new runbook [[frank-npu]] (linked from both indexes),
|
||
AGENTS.md fleet table + runtime section updated with the IOMMU/SVA hard
|
||
requirement. Client wiring: pi `models.json` registers the endpoint under the
|
||
Dankclaw provider with the exact FLM registry tag as model id.
|
||
|
||
## 2026-08-30 (night) — frank fleet double swap: STRIX on its pinned fork, Huihui on FreeToken
|
||
- **Dankclaw / frank-large**: swapped to
|
||
`kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF` on the card's
|
||
pinned `kingjones30/ROCmFPX` runtime (built at
|
||
`/home/dank/GitPackages/kingjones-ROCmFPX/build`, `LLAMA_BUILD_WEBUI=OFF`
|
||
— npm/HF-bucket asset provisioning aborts the link at 98% otherwise).
|
||
On-host 113.47 GiB file verified byte-identical to the HF LFS oid
|
||
(no re-download). Kept our args (401408×2, ub 512, fa on, fit off);
|
||
fa-on works UNPATCHED on this model — the QSA F32 assert was
|
||
agentionai-specific. No spec decode (card ships none; MTP heads
|
||
unloadable). Validated live: ~225 s load, health 200, reasoning
|
||
completions, real 111k-token prefill at 213–232 t/s. Deployed via
|
||
git flow; manual-process race during validation caught and fixed
|
||
(unit was crash-looping on a bound port while its health check
|
||
passed — check MainPID, not just /health).
|
||
- **Dankserver / frank-small**: swapped the model within
|
||
FreeToken v0.1.2 to `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8`
|
||
(native block-fp8, same loader path and pool plan; 35 GB downloaded
|
||
via `hf`, official checkpoint retained for rollback). Learned:
|
||
FreeToken's chat API 422s without an explicit `model` field. ~40
|
||
tok/s verified post-deploy. Both `.service` Descriptions refreshed.
|
||
- Docs: [[kingjones-strix-rocmfp4]] reactivated as the large runbook
|
||
(runtime build + swap procedure + validation), [[agentionai-fast-imatrix]]
|
||
marked RETIRED (rollback record), [[freetoken-frank-small]]
|
||
model sections updated, LLM index + AGENTS fleet tables refreshed.
|
||
|
||
## 2026-08-30 (late) — download chain proven end to end; qB credentials persisted
|
||
- **Cracked the arr↔Prowlarr feed contract**: route is `/{indexerId}/api`
|
||
(Prowlarr `NewznabController`; absolute `/api/v1/indexer/{id}/newznab`),
|
||
auth = the Prowlarr **global** key. The previously hand-rolled
|
||
`/torznab/{id}/feed` paths were Forms-auth catch-alls returning
|
||
`302 → /login` — which the arrs XML-parsed as "unexpected token '/'".
|
||
A source-verified librarian answer settled it in one shot.
|
||
- **Re-pointed all four arr indexer rows** (Radarr→YTS id7,
|
||
Sonarr/Lidarr→Nyaa id4, Readarr→Byrutor id3) via each app's own API;
|
||
every PUT passed the app's live connection validator (202); RSS enabled.
|
||
- **TV + music chains proven** by live search: Sonarr returned 76 current
|
||
releases via Prowlarr-Nyaa; Lidarr initially logged `0 active indexers`
|
||
(new rows default all-off) — enabling the search flags yielded 18
|
||
releases for YOASOBI – THE BOOK.
|
||
- **First real grab**: `MoviesSearch` → 3 YTS releases → *The Dark Knight
|
||
(2008) 2160p* (7.5 GiB) → qBittorrent 100 % → **imported**
|
||
(`hasFile: true`, 8,078,103,833 B at `/movies/The Dark Knight (2008)`,
|
||
hardlinked from the shared `/mnt/entropy` filesystem), queue empty.
|
||
- **Prowlarr Application auto-sync found broken by version skew**
|
||
(`Unable to cast object of type 'NzbDrone.Core.Thing…'` against Radarr
|
||
6.3) — the manual rows are the integration; documented.
|
||
- **qBittorrent permanent credentials installed**: the WebUI temp password
|
||
rotated on every boot (it had silently broken the arr clients once this
|
||
session). qb5 removed the API password endpoint and the lscr image has
|
||
no env support — wrote `WebUI\Password_PBKDF2` (source-verified
|
||
`base64(salt):base64(PBKDF2-HMAC-SHA512/100k)` form) into the conf with
|
||
the container stopped; login survives `podman restart` (204). All arr
|
||
download clients updated to it; value kept host-local in
|
||
`/home/dank/.qbpass` (0600), never in git.
|
||
- **Readarr parked with reason**: abandoned upstream; its qB client cannot
|
||
authenticate against qB 5 while three sibling apps pass with identical
|
||
credentials. Indexer row healthy; books pipeline awaits a successor.
|
||
- **Docs**: rewrote `Dankserver/prowlarr.md` (live wiring + six verified
|
||
gotchas incl. the joiner bug and the 302-masquerading-as-auth failure)
|
||
and `Dankserver/qbittorrent.md` (persistent-credential recipe, Referer
|
||
CSRF note).
|
||
|
||
## 2026-08-29 (late) — guarded-host sign-in fixed end to end
|
||
- **Diagnosed** both user symptoms: `Invalid next URL` = the flow being entered with an absolute cross-host `next=` (open-redirect guard rejects at completion); phone "hostname could not be found" = the public DNS zone had **no `auth` A record** (added via Porkbun API, `auth A 71.227.227.172 TTL 300`).
|
||
- **Disproved meta-header routing** live (full meta-header set still 404s; matching `Host` alone answers). The 2026.8 embedded outpost routes solely on the original `Host` vs each proxy provider's `external_host`.
|
||
- **Provisioned** proxy providers + applications for `search`, `hermes`, `prometheus`, `romm`, `files` (pks 2–6) via ORM, attached to the embedded outpost; `internal_host = http://caddy:8017` everywhere.
|
||
- **Wedge found + fixed live:** providers created with null `authorization_flow`/`invalidation_flow` starve the outpost's config fetch (`Failed to fetch outpost from API, retrying in 3 seconds` forever → entire outpost URL space 404s). Backfilled all three flows; the 3-second retry loop recovered without restart.
|
||
- `/dankclaw/Caddyfile` (`29f9a86`) — `:8016` is now a uniform whole-request `reverse_proxy authentik-server:9000` preserving `Host`; deleted the per-vhost redirect/flow design and stale comments.
|
||
- `/dankclaw/anubis/policy.json` (`2a72efa`) — ALLOW `^/outpost\.goauthentik\.io/` so login/transport paths are reachable before the challenge dance; app paths remain challenged.
|
||
- **Gotcha:** the policy bind-mount pins the file inode; a git checkout of a new policy file does NOT reach the running distroless container (observed 22 h of stale policy with zero reload logs). `--service anubis` deploy restart is mandatory after policy edits.
|
||
- `/tests/authentik.bats` — meta-header test replaced by the upheld contract (Host preserved; no absolute cross-host `next=` anywhere in the Caddyfile). Suite green (`0` not-ok).
|
||
- `/docs/Infrastructure/Authentik.md`, `/docs/Infrastructure/Anubis.md`, `/docs/Architecture/Network Architecture.md` (`ee1e038`) — rewritten to the live-verified mechanism: Host→provider routing, lockstep rule (guard set = TLS edge = :8017 table = provider list = public DNS), provisioning pitfalls, failure-mode rows for both symptoms.
|
||
- **Verified** by headless JS client: `https://search.franksdank.cloud/` → Anubis PoW solved → outpost intercept → authentik login form renders with the return path inside the signed OAuth2 `state` JWT. Credential submission is the user's step.
|
||
|
||
## 2026-09-14
|
||
- **Creation**: [[Hermes-WebUI]] — new `hermes-webui` compose service on Dankclaw (the Hermex server): runs `hermes-webui`'s `bootstrap.py --foreground` in the agent image, shares `/home/dank/hermes-data` with the gateway (identical rootless Podman uid mapping), publishes `127.0.0.1:8787` only, password-gated via `HERMES_WEBUI_PASSWORD` in the host `.env`.
|
||
- `/dankclaw/compose.yaml`, `/dankclaw/.env.example` — service definition and password placeholder; container-internal bind is `0.0.0.0` (Podman's port forwarder dials the bridge IP) while the host-side publish stays loopback so the SSH tunnel is the sole ingress.
|
||
- `/scripts/deploy-plan.py`, `/scripts/deploy.sh`, `/scripts/deploy-remote.sh` — `hermes-webui` registered as an isolated Dankclaw service in all three inventories; planner narrows edits to its compose block only.
|
||
- `/tests/hermes_webui.bats` — new seam: foreground entrypoint, loopback publish + auth, shared volume, `/health` healthcheck, env placeholder, planner narrowing (hermetic fixture repo), deployer dispatch.
|
||
- Remote access pattern documented: iPhone Termius local forward `127.0.0.1:8787 -> 127.0.0.1:8787` over the existing router port forward to Dankclaw `:22`; Hermex points at `http://localhost:8787`. No Tailscale, no Cloudflare.
|
||
|
||
## 2026-08-26
|
||
- `/dankclaw/Caddyfile` — Both Plex vhosts now proxy to PMS over a TLS upstream (`insecure_skip_verify`); PMS drops plain-HTTP connections from non-loopback sources, which produced empty-body 502s.
|
||
- `/docs/Infrastructure/Caddy.md` — Documented the Plex TLS-upstream requirement and the `/web/` vs `/` distinction.
|
||
- `/tests/caddyfile.bats` — Added a test pinning the TLS upstream transport on both Plex vhosts.
|
||
- `/docs/Dankclaw/Gitea.md` — Documented migration of Gitea web proxy ownership to canonical Caddy and retirement of the standalone proxy.
|
||
- `/docs/Dankclaw/Dankclaw Stack.md` — Updated boot-persistence and Caddy deployment descriptions for the two-unit Gitea stack.
|
||
- `/docs/Architecture/Deployment Flow.md` — Replaced the split-proxy safety description with the canonical Caddy ownership model.
|
||
- `/dankclaw/gitea/quadlet/gitea-caddy.container` — Retired the standalone Gitea Caddy Quadlet; deployment stages any live copy into home trash.
|
||
- `/scripts/deploy-plan.py` — Routes retirement of the legacy proxy to the canonical Caddy and removes it from the service inventory.
|
||
- `/scripts/deploy.sh` and `/scripts/deploy-remote.sh` — Removed standalone proxy dispatch and added idempotent retirement before canonical Caddy deployment.
|
||
- `/tests/caddyfile.bats`, `/tests/deployment_scoping.bats`, `/tests/deployment_flow.bats`, `/tests/frank_large_config.bats` — Added migration coverage and corrected the existing ubatch harness assertion.
|
||
- `/docs/index.md` — Bundle index remains the root navigation entry for the updated pages.
|
||
- **Boot-persistence audit (post-reboot, 2026-08-26 evening):** after Frank's reboot, Caddy and Pi-hole were both down. Caddy ran as a bare compose container (`restart: unless-stopped` does not survive shutdown's explicit stop; rootless Podman has no boot-time auto-start for bare containers). Pi-hole's quadlet existed but crash-looped (restart counter 1645): `net.ipv4.ip_unprivileged_port_start` was reset to 80 by two sources — a later-sorted sysctl drop-in (`99-rootless-ports.conf`) and a stale line in `/etc/sysctl.conf`, which systemd-sysctl applies **after** all of `sysctl.d/` (the first deploy pass only fixed the drop-in; the runtime value stayed 80 until the `/etc/sysctl.conf` line was neutralized too).
|
||
- `/AGENTS.md` — new MANDATORY section "Boot Persistence & Self-Healing": every deployed, proven-working container must have a self-healing, boot-persistent quadlet unit (`WantedBy=default.target` + enabled + linger, `Restart=always`, repo-provenance `.container`), verified before a service is declared deployed.
|
||
- `/dankclaw/caddy/quadlet/caddy.container`, `/dankclaw/caddy/env.example` — Caddy is now a boot-persistent quadlet (`caddy.service`): joins `dankclaw_default` so container-name upstreams keep resolving, `Pull=never` (local-only image), Porkbun credentials via `/home/dank/Containers/caddy/env` (deploy writes it from `.env`), bare-IP-catch-all health check. The caddy service was removed from `/dankclaw/compose.yaml`.
|
||
- `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` installs it to `/etc/sysctl.d/`, stages conflicting drop-ins (value > 53) in the home trash, comments out a stale `/etc/sysctl.conf` line (applied after all of sysctl.d), and applies `sysctl --system` (the boot code path).
|
||
- `/scripts/deploy-remote.sh` — `deploy_caddy` installs/starts the quadlet (retiring the bare container; all state is in host volumes). Boot persistence comes from podman's quadlet generator, which creates the `default.target.wants` symlink from the unit's `[Install]` section at every user-manager start — `is-enabled` reports `generated` for all healthy quadlets, and an explicit `systemctl enable` fails on generated units (learned live during the deploy). `deploy_pihole` enforces the sysctl prerequisites.
|
||
- `/tests/caddy_quadlet.bats` (new seam), `/tests/caddyfile.bats`, `/tests/deployment_flow.bats`, `/tests/pihole.bats` — quadlet boot-persistence, self-healing, network, env, and sysctl coverage; compose assertions repointed from the removed caddy service to the quadlet.
|
||
- `/docs/Infrastructure/Caddy.md`, `/docs/Infrastructure/Pi-hole.md`, `/docs/Dankclaw/Dankclaw Stack.md` — quadlet runtime facts, the 2026-08-26 incident, and the sysctl-conflict failure mode.
|
||
|
||
## 2026-09-01
|
||
- dankserver SSH recovered after the reboot (the "Not allowed at this time" port-22 responder vanished with it); the media stack had been restarted bare ~35 min post-boot, then the quadlet cutover completed for all six services (romm trio 13:45, jellyfin 13:49, calibre-web 13:50, copyparty 13:53 PDT) — units generated from the repo, data roots migrated to `/home/dank/Containers/<svc>`, `/opt/{romm,jellyfin,calibre-web,copyparty}` retired, all four endpoints answering, legacy `container-*` units gone. Dankserver deploy-state for the six services records deployed revisions through `1a257c0`.
|
||
- `/scripts/deploy-remote.sh` — frankbook handler fix: the credential-gate SKIP state now propagates Mac-side before restart+verify (a parked qbittorrent previously failed the batch and aborted the five arrs); qB first-boot seed writes a PBKDF2 `@ByteArray` WebUI password (plaintext is ignored by qB ≥ 4.2), sets `HostHeaderValidation=false` + `ServerDomains=*` for the socat forwarder path, and pre-creates the four categories with `save_path=/Volumes/COMPLETE/Finished Torrents`.
|
||
- `frankbook-pro` — full 7-service deploy run: gluetun + qbittorrent SKIP cleanly (WireGuard credentials pending — the Mac needs its own peer; reusing Dankclaw's key would flip its tunnel), prowlarr/sonarr/radarr/lidarr/readarr deployed, healthy, and reachable from dankserver over the LAN via the socat LaunchAgents (192.168.0.112).
|
||
- jellyfin GPU posture on dankserver: `AddDevice=/dev/dri` + `GroupAdd=keep-groups` in the unit (parity with pre-migration reality — no `/dev/nvidia*` in the container, no working VA-API driver in the image; the old bare container had the same. CPU/direct-play serves the household).
|
||
- `/docs/Infrastructure/FrankBook-Pro.md` (new), `/docs/index.md` — documented the Mac stack: VM placement, WireGuard gate, deploy flow, socat LaunchAgents, and the qBittorrent PBKDF2 seed.
|
||
- `/dankclaw/seerr/quadlet/seerr.container` — New Seerr (seerr-team/seerr v3.4.1) request-manager unit on Dankclaw: loopback-only publish 127.0.0.1:5055, SQLite under /home/dank/Containers/seerr/data, on dankclaw_default for the post-auth route.
|
||
- `/dankclaw/Caddyfile`, `/dankclaw/pihole/addn-hosts` — seerr.franksdank.cloud vhost (Anubis guard + :8017 @seerr post-auth handler) and LAN DNS record, deployed in lockstep with the Authentik outpost provider `seerr-proxy` (external_host https://seerr.franksdank.cloud/, internal_host http://caddy:8017) and the Porkbun A record 192.168.0.143.
|
||
- `/scripts/deploy-remote.sh` — deploy_seerr handler (install_quadlet + pre-create bind dir + restart).
|
||
- `/tests/seerr_quadlet.bats` — new seam: boot persistence, self-healing, pinned tag, loopback-only publish, data-root contract, appnet join, no env/envfile secrets.
|
||
- `/serverAppConfigs-docs/Dankclaw/Seerr.md`, `/serverAppConfigs-docs/index.md` — service doc + index entry.
|
||
- Lessons: rootless podman uid mapping is continuous (container N → host 100000+N-1; uid 1000 → 100999 — an off-by-one chown EACCES-crash-looped first boot); new quadlet bind-source dirs must pre-exist on target (statfs 125 otherwise); new vhost TLS issues at Caddy reload (~15 s dns-01), not at first request.
|
||
- Authentik outpost claim cache fixed (root cause of the persistent `/admin` 403 despite homepage-admins membership): the outpost serves per-browser-session claims from `authentik_providers_proxy_proxysession` until row expiry — akadmin's live row predated the group add and carried `["authentik Admins"]` only, and the 403 never generates an authentik event. Deleted his 4 rows; the next visit re-authorizes silently against the untouched authentik login session. Silent re-auth proven live: deleted qa-e2e-admin's 13 rows, ran the e2e admin spec against the stale storage state — 2 passed in 2.9 s, no login prompt. Mechanism + diagnosis SQL documented in `/serverAppConfigs-docs/Infrastructure/Authentik.md` (new "Outpost claim cache" section + failure-mode row); the doc's data-path references updated to the 2026-09-01 Containers layout.
|
||
- WireGuard credentials provisioned (Panama PA-12 conf → gluetun env inside the VM): tunnel egress verified (exit IP Panama, not residential), qB WebUI published per plan, and Prowlarr → qBittorrent wired — all five download clients (prowlarr, sonarr, radarr, lidarr, readarr) test 200 against live qB through the podman gateway `10.88.0.1:8080`; Prowlarr's client created via its `/downloadclient/schema` template (hand-minimal body 400s on create-time test).
|
||
- `/frankbook-pro/qbittorrent/quadlet/qbittorrent.container` — pinned `lscr.io/linuxserver/qbittorrent:4.6.7`: `latest` had resolved to qB 5.x, whose `204` empty-body login success Readarr's retired client rejects as "Authentication failure" at runtime (not just in API creates — the prior session's "runtime cookie auth unaffected" assumption was wrong); 4.6.7 answers the legacy `200` + `"Ok."`. Gotcha: after 5.x ran, its conf rewrite left a PBKDF2 line 4.6.7 rejects — every login `invalid credentials`; rewriting the same line fresh (same password, 100k iterations) fixed it.
|
||
- `/docs/Infrastructure/FrankBook-Pro.md` — wiring, pin rationale, PBKDF2 version-flip gotcha, and E2E result (test torrent through the Panama tunnel at ~17 MB/s, INCOMPLETE → COMPLETE into `Finished Torrents`, deleted with data).
|
||
- `/scripts/deploy.sh` + `/scripts/deploy-plan.py` + `/tests/dankserver_quadlet.bats` — committed earlier today (`b206f47`): dankserver quadlet standardization units/handlers/routing.
|
||
|
||
## 2026-09-04 — Sonarr moved to the admin side; internal-only by construction
|
||
- Operator directive (fixing an earlier seerr/sonarr mix-up): Sonarr is an
|
||
**admin-side** service and must be locally reachable only. The homepage
|
||
registry flipped its card from the user category to **admin** and repointed
|
||
the link at `http://192.168.0.112:8989` (LAN-direct). Users keep requesting
|
||
TV through Seerr, which stays on the user side behind the guard chain.
|
||
- Exposure closure: the unguarded public vhost `sonarr.franksdank.cloud`
|
||
(direct `reverse_proxy 192.168.0.112:8989`, no auth) and its post-auth
|
||
`@sonarr` handler were retired from the Caddyfile; the Pi-hole addn-hosts A
|
||
record for the name was dropped with it. No public Porkbun record ever
|
||
existed (NXDOMAIN at the authoritative zone), so after this change there is
|
||
no WAN path to Sonarr at all — manual DNS could previously reach the vhost,
|
||
which served the zero-auth UI to anyone who resolved the name manually at
|
||
the public edge; the vhost (added in `e0bade6`, 2026-09-04) is retired.
|
||
|
||
## 2026-09-15/16 — frank-large: ik_llama.cpp cutover + the decode campaign
|
||
- Deployed (0c1e10a, via clean-worktree stash cycle around the peer's
|
||
uncommitted files): halogen quadlet retired to
|
||
`~/.AGENTTRASH/20260915-ik-swap/halogen-quadlet`, frank-large now a root
|
||
systemd unit (`scripts/frank-large.service` → `/etc/systemd/system/`)
|
||
running the committed serve script. Service verified live: health 200,
|
||
reasoning completions, vision, slot persistence.
|
||
- Overnight performance campaign against the halogen bar (Frank: ~30 t/s
|
||
decode). Result: decode stuck at 3.8-4.3 t/s across every safe
|
||
configuration; prefill ~192 t/s = the CPU AVX512-VNNI ceiling. Full
|
||
evidence chain in [[frank-large-ik-uncensored]]: (1) ~100 CPU↔GPU backend
|
||
splits per token with the 3090 at 5% SM utilization — the decode wall is
|
||
GPU-section launch/sync overhead, not compute or memory placement;
|
||
(2) T1 ≈ T16 decode (3.97 vs 3.8-4.0) — the wall is a serial critical
|
||
path; (3) ik's CUDA-graph capture aborts on qwen4exp's per-layer
|
||
`ggml_repeat_4d` hc-broadcast ops; (4) mainline llama.cpp `37b53fd` runs
|
||
1.0 t/s on this model (CPU MoE kernels 4x slower than ik's iqk) — ruled
|
||
out; (5) the GGUF has no MTP head (mradermacher stripped it) — no
|
||
speculative decoding; (6) 84 GiB of experts cannot fit 24 GiB VRAM, so
|
||
halogen's all-resident 30 t/s is physically out of reach for this model
|
||
on this host. Re-evaluate when mainline's hc-ops + graph handling mature
|
||
(ggml-org PR #28901).
|
||
/- Measurement integrity traps hit live: the systemd unit's Restart=always
|
||
respawned pkill'd servers mid-experiment (phantom instances answering
|
||
health — manual experiments must `sudo systemctl stop frank-large.service`
|
||
first; plain `systemctl --user` lines are stale halogen-era docs); a
|
||
concurrent `-j16` build starved a decode measurement into a false timeout;
|
||
warm-cache `cat` benchmarks invalidated a concurrent perf window.
|
||
- `-ncmoe 42` (+6 GPU expert layers) measured 4.54 t/s decode at 22.0/24.6
|
||
GiB VRAM — documented as a tuning lever, not shipped (OOM margin at 262k
|
||
ctx is operator's call). `-ncmoe 48 -thp` at 4.6 t/s holds a 110 GiB
|
||
anonymous host buffer — co-tenant killer, never ship.
|
||
- Trash staged for approval: dankclaw `~/.AGENTTRASH/20260916-ik-partfiles/`
|
||
(111 GiB, the 3 GGUF download parts, superseded by the concatenated
|
||
verified GGUF), `~/.AGENTTRASH/20260915-ik-swap/` (halogen quadlet),
|
||
repo `tests/.AGENTTRASH/frank_large_quadlet.bats` (retired seam),
|
||
`scripts/.AGENTTRASH/frank-small*` (earlier retirement).
|
||
- Docs: `serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md` (new
|
||
runbook), LLM + root indexes relinked, AGENTS.md topology row +
|
||
Active Services section + bench-policy unit type. Committed 795b0b3;
|
||
tests 300 ok with the single pre-existing not-ok-73 (comfyui selector
|
||
bug, unrelated).
|
||
|
||
## 2026-09-16 — Dankclaw re-split: halogen returns as frank-large, llama.cpp takes the 3090 as frank-small
|
||
- `/dankclaw/frank-large/quadlet/frank-large.container` — halogen-flash-server
|
||
restored as the active runtime (the 09-15 ik swap never removed the file) and
|
||
retuned to the shape Frank asked for: `HALOGEN_CTX` 262144 → **200192**,
|
||
`HALOGEN_KV_POOL_POSITIONS` 401408 → **400384** (exactly 2 × 200192),
|
||
`HALOGEN_KV_SLOTS` 3 → **2**, `HALOGEN_MAX_TOK` 32768 → **24576** and
|
||
`HALOGEN_HOST_RESERVE_GIB` 15 → **20** (arena −4.2 GiB, for the co-tenant),
|
||
`HALOGEN_VISION_TOWER` **unset** (the tower moved to frank-small). Live ledger:
|
||
budget `121 − 67.7 − 20 = 33.5 GiB` against a need of `27.2`, so **no clamp**;
|
||
68.0 GiB weights + 11.0 KV + 16.5 working = 95.4 GiB held, 14.7 GiB left for
|
||
everything else (475 contiguous 2 MiB blocks). The co-tenant stall NOTE is
|
||
back, and the boot logged 4,350 compaction stalls (285 failed) — tight and
|
||
fragmented, not out of memory.
|
||
- `/scripts/frank-small.env`, `frank-small.service`, `serve-frank-small.sh` (new)
|
||
— mainline llama.cpp `37b53fd` on the RTX 3090 serving
|
||
`ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF` at IQ3_S: the `-mtp` build (the head is
|
||
`blk.64.nextn.*` in the SAME file, so `--spec-type draft-mtp` runs the draft
|
||
context on the main model's weights — no second copy, no `-md`) plus the BF16
|
||
projector, 131072 ctx / 1 slot, f16 KV, `--jinja` and `--metrics` pinned, port
|
||
7771, root systemd unit. 22,362 MiB of the card's 24,576 in use. Verified:
|
||
`/health`, chat with `reasoning_content`, a `tool_calls` round trip, slot
|
||
timing `draft acceptance 0.66–0.94`, and an image answered from the projector.
|
||
- `/scripts/deploy-remote.sh` — `deploy_llm_large` flipped back to halogen
|
||
(retire the ik ROOT unit into
|
||
`~/.AGENTTRASH/20260916-halogen-restore/host-units`, pull + re-tag the image,
|
||
**`wait_port_free 7770`**, install the Quadlet, start) and `deploy_llm_small`
|
||
added (root unit, model-file gate, health on 7771). The port gate exists
|
||
because the FIRST cutover attempt failed exactly there: pasta reported
|
||
`Address already in use` about a second after the ik server exited, the unit
|
||
died with code 126, and only `Restart=on-failure` recovered it. A redeploy now
|
||
stops this service's own quadlet before waiting, or the gate would deadlock
|
||
against itself.
|
||
- `/scripts/deploy-plan.py`, `/scripts/deploy.sh`, `/scripts/test-changed.sh` —
|
||
frank-small added to `SERVICE_ORDER` and the Dankclaw inventory (the stale
|
||
`dankclaw:frank-npu` entry dropped with it); the changed-test mapping for
|
||
`dankclaw/frank-large/*` pointed at `tests/frank_large_quadlet.bats`, retired
|
||
on 09-15, so the wrapper had been failing on that path ever since — now
|
||
`tests/frank_large_unit.bats`.
|
||
- Retired to `scripts/.AGENTTRASH/`: the ik trio (`frank-large.env`,
|
||
`frank-large.service`, `serve-frank-large.sh`), `chat-template.jinja` (only the
|
||
ik runtime read it), and four benchmark harnesses bound to retired runtimes.
|
||
- `/tests/frank_large_unit.bats` rewritten for the Quadlet (the `pool == 2 × ctx`
|
||
invariant, no vision tower, the port gate) and `/tests/frank_small_unit.bats`
|
||
added (root-unit shape, MTP-from-the-file contract, KV budget vs the card, port
|
||
separation). One assertion failed on its first run because the negative grep
|
||
matched a COMMENT in the serve script — the test now strips comments before
|
||
grepping the effective flags.
|
||
- `shared/homepage/services.json`, `shared/prometheus/prometheus.yml`,
|
||
`rules/llm-fleet.yml`, `grafana/.../llm-fleet.json` — a frank-small card at
|
||
`127.0.0.1:7771` and a `frank-small` scrape job at `192.168.0.143:7771`; the
|
||
`frank-large` job **removed**, because the halogen engine has no `/metrics` at
|
||
all (its routes are `/health`, `/cache` and `/v1/*`), so a target there would
|
||
have been red by construction; the dashboard's endpoint picker had been sourced
|
||
from a vLLM-only metric, so a llama.cpp job could never be selected. Live: 11
|
||
targets, 7 up, 4 down (comfyui, gitea, services — all structural), and all
|
||
three LLM homepage cards up.
|
||
- Harness (`~/.omp/agent/models.yml` + `config.yml`, dev box) — third provider
|
||
`llamacpp (dankclaw-3090)` for `:7771`; `frank-large` ctx 262144 → 200192 and
|
||
`input: [text]`; roles `smol`, `tiny`, `task`, `vision` → frank-small (the
|
||
image path and the small/fast roles), `slow`/`plan`/`designer`/`advisor` →
|
||
frank-large. Two failures found by driving it, both fixed: (1) frank-small's
|
||
template accepts only `xhigh|medium|low` for `reasoning_effort` and 500s on
|
||
anything else, and subagents run at `high` — every subagent on that endpoint
|
||
died until `compat.reasoningEffortMap` folded the levels onto that set; (2) the
|
||
ik runtime that had been serving `frank-large` rejected every request carrying
|
||
`tools` (`500 tools param requires --jinja flag`), which is what killed a
|
||
monitoring subagent earlier in the day. Both endpoints now answer tool calls,
|
||
and a subagent completes a tool-using turn.
|
||
- Docs: `frank-small-qwen38-gsq-rco.md` (new runbook),
|
||
`halogen-flash-server.md` re-activated with the 09-16 ledger,
|
||
`frank-large-ik-uncensored.md` marked retired, LLM index, deployment flow,
|
||
Observability and AGENTS.md topology/sections/lessons. Three AGENTS.md defects
|
||
found on the way: a truncated sentence mid-file ("There is **no audio"), a
|
||
harness path that does not exist (`/home/dank/.omp/agent/models.yml` — the
|
||
harness runs on the dev box), and the benchmarking stop/mask recipe still
|
||
describing frank-large as a root unit.
|
||
- Deploy: `./scripts/deploy.sh dankclaw --service frank-large --service
|
||
frank-small` — per-service health gates, then the e2e gate PASSED (31 passed,
|
||
1 flaky-retried). Suite: 323 seams ok, 0 failures.
|
||
|
||
## 2026-09-16 — The x4 pair: measured what a 3090 + Strix Halo combination can actually do
|
||
- Built mainline llama.cpp `37b53fd` with **both** the CUDA and Vulkan backends
|
||
(`build-dual`; glslc and libvulkan were already installed), to test the design
|
||
the earlier analysis only sketched: trunk + KV on the 3090, MoE experts on the
|
||
iGPU, PLE in LPDDR, one hop per boundary.
|
||
- Device map found: `CUDA0` = the 3090, `Vulkan0` = the same 3090, and
|
||
**`Vulkan1` = the iGPU** (RADV STRIX_HALO, `uma: 1`, 114 GiB GTT). The iGPU
|
||
being `Vulkan1` and not `Vulkan0` is exactly the detail that silently measures
|
||
the wrong device.
|
||
- Tensor inventory of the 111 GiB GGUF (mradermacher Flash-Next Uncensored):
|
||
experts **75.0 GiB**, `per_layer_token_embd` **32.8 GiB**, everything else
|
||
**3.2 GiB**; 12 of 48 layers are full attention (`[3,7,11,…,47]`), so only
|
||
they carry KV — at the engine's 28 KiB per position that is 5.5 GiB of KV read
|
||
per token at 200k context.
|
||
- Results (llama-bench, both services stopped per policy): iGPU-only (Vulkan,
|
||
PLE on CPU) **32.64 tok/s at ~0 depth, 25.77 tok/s at 64k depth, pp 456.7 →
|
||
231.8 into a 64k cache**, with the iGPU at 99–100 % busy; iGPU trunk + CPU
|
||
experts 12.10; 3090 trunk + CPU experts 20.46; and the cross-backend design
|
||
**aborted in the scheduler** — `pre-allocated tensor
|
||
(blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the
|
||
operation (NONE)` — while both per-layer cross-backend forms failed to load.
|
||
Mainline cannot serve one model from CUDA + Vulkan; the CPU is the only
|
||
fallback side that works.
|
||
- The comparison that matters: at fleet-realistic depth the iGPU-only run
|
||
matches halogen's field ~25 tok/s with the 3090 **idle** — but it commits 116
|
||
of the host's 121 GiB (78 GiB of Vulkan device buffers + 32.8 GiB of PLE page
|
||
cache) against halogen's 95.4 GiB held / 14.7 GiB spare, so it is not
|
||
deployable beside the fleet's ~28 co-tenants. Throughput parity, memory
|
||
bankruptcy.
|
||
- Also recorded: `GGML_CUDA_NO_PINNED` is mainline's pinning guard (ik called it
|
||
`GGML_CUDA_NO_PINNED_WEIGHTS`); `systemctl mask` is refused for a relinked root
|
||
unit (frank-small's unit is a symlink into the repo); and llama-bench's `tg`
|
||
instances run with `n_prompt = 0`, so `-d` is required to put KV into the
|
||
measurement at all. Both services were stopped and restored (both healthy;
|
||
halogen's ledger read `400384 positions, ~27.2 GiB` against a 33.5 GiB budget
|
||
with 15.2 GiB left).
|
||
- New page: [[x4-pair-experiment]].
|
||
|
||
## 2026-09-16 (later) — the x4 pair, round two: the first conclusion was a parsing artifact
|
||
- Asked whether **pure Vulkan** changes the answer (both devices can run Vulkan —
|
||
`Vulkan0 = RTX 3090 (NV_coopmat2)`, `Vulkan1 = RADV STRIX_HALO (KHR_coopmat)`).
|
||
It does not: the per-tensor design still aborts, byte-identically.
|
||
- **The real gate is `-ot`, not the driver mix.** `has_tensor_overrides()` is a
|
||
required term of `pipeline_parallel` in `llama-context.cpp:433` — the ONLY place
|
||
in the tree that reads it — and that flag is what sets `n_copies` in
|
||
`ggml_backend_sched_new` (`ggml-backend.cpp:25`). With `n_copies == 1` a
|
||
pre-allocated weight on a non-primary device has no backend, so the graph
|
||
aborts at `ggml-backend.cpp:941` with the leaf's own op name (`(NONE)`).
|
||
`-ot exps=CPU` works only because the CPU backend is last in the list.
|
||
- **Then the correction.** `llama-bench --help` says `-dev <dev0/dev1/...>`: `/`
|
||
splits a device LIST, `,` splits SWEEP values. So `-dev CUDA0,Vulkan1` was two
|
||
*single-device* runs — the earlier "CUDA + Vulkan cannot serve one model", the
|
||
"load failures", and the OOMs were artifacts (the OOM was 111 GiB on the 24 GiB
|
||
card). The bench table's `dev` column had been printing the truth all along.
|
||
- **With the correct syntax two-device splits work, in both flavours** (64k depth,
|
||
`-d 65536`; placement proven by `nvidia-smi` sampled during the run):
|
||
- pure Vulkan `-dev Vulkan1/Vulkan0 -ts 0.85/0.15` — 12.8 GiB on the 3090,
|
||
**27.55 tok/s** (pp512@64k 224.3)
|
||
- cross-vendor `-dev CUDA0/Vulkan1 -ts 0.15/0.85` — 15.0 GiB on the 3090,
|
||
27.23 tok/s with **pp512@64k 268.5**
|
||
- against the iGPU alone: 25.77 tok/s / 231.8 — so **+7 % decode, +16 % prefill**
|
||
- **And the split still cannot reach the design's premise**: whole layers move
|
||
their experts with them (1.62 GiB per layer), the 12 attention layers are spread
|
||
every 4th across the depth, and the 32.8 GiB PLE must stay on the main device —
|
||
so KV cannot be consolidated on the 3090.
|
||
- **Incident, same evening:** the rootless podman pause process died — trigger
|
||
NOT established (no OOM kill in `journalctl -k` for that window, so "the
|
||
116 GiB runs squeezed it out" is a guess, not a finding). Every `podman`
|
||
command then failed (`invalid internal status … could not find any running
|
||
process`), every container start failed on aardvark-dns binding `10.89.x.1:53`
|
||
(the rootless netns holding the bridges was gone), and all 28 quadlets looped.
|
||
`podman system migrate` — podman's own named remedy — **panics** in 5.7.0
|
||
(`teardownNetworkBackend`). Repair was a **reboot**: all 28 containers returned
|
||
on their own (linger + `WantedBy=default.target` present), e2e gate green after.
|
||
- **A second, pre-existing defect surfaced by that reboot:** `hermes.franksdank.cloud`
|
||
returned 502. hermes-webui is `Network=host`, so 8787 is a bare host-netns bind
|
||
and netavark DNATs only *published* ports — from inside the Caddy container
|
||
`192.168.0.143:8787` and `10.89.1.1:8787` are refused while
|
||
`host.containers.internal:8787` (169.254.1.2) answers 200. Fixed in the
|
||
Caddyfile, the stale claim in its comment replaced with the measurement, and the
|
||
seam that guarded it **strengthened** (it accepted any numeric upstream, so the
|
||
LAN IP had been passing it; reverting now fails it — teeth verified).
|
||
- Both endpoints restored and healthy: frank-large `400384 positions, ~27.2 GiB`
|
||
against a 33.5 GiB budget (no clamp), frank-small 200 on 7771. Page:
|
||
[[x4-pair-experiment]] (rewritten).
|
||
|
||
## 2026-09-16 (night) — model cache cleared on instruction, then re-fetched, and the update was real
|
||
- **Operator instruction:** purge the model cache outright so no run can serve a
|
||
stale payload. Done with one-time `rm` authorisation, and the two *live*
|
||
payloads were cleared and immediately re-fetched — because deleting them
|
||
without a re-fetch means neither endpoint can start.
|
||
- **Cleared:** the Uncensored i1-Q4_K_M payload (112G), the Q4_K_S part-fragments
|
||
(105G), the retired kingjones STRIX and agentionai ROCmFP4 payloads (114G +
|
||
863M), an unreferenced Qwen3.5 vision dir (8.8G), and all of `~/.AGENTTRASH`
|
||
including the ik part-files (119G). Disk went **298 GB → 895 GB free**; 814 GB
|
||
after re-fetching what the services need.
|
||
- **The re-fetch produced genuine updates.** halogen's sidecars had been stale —
|
||
`overlay.hgn` 2,477,677,120 → **2,572,466,560** B, `overlay-speed.hgn`
|
||
2,383,306,048 → **2,478,095,488** B — while `w4b.hgn` came back the same size
|
||
(124,068,083,904). So "clear and re-pull" *is* an update mechanism here, and
|
||
the sidecars are versioned **in place**: a stale copy is invisible from the
|
||
file name. frank-small's 27B + mmproj re-fetched to hashes matching the Hub
|
||
exactly and are unchanged.
|
||
- **Trap:** podman refuses a bind mount whose source directory is missing
|
||
(`Error: statfs /home/dank/models/halogen: no such file or directory`), so
|
||
frank-large looped on `Restart=on-failure` until `mkdir -p` recreated the
|
||
**empty** directory — then the engine fetched the repo itself
|
||
(`HALOGEN_DOWNLOAD`), which is the documented refresh path.
|
||
- **Q4_K_S vs Q4_K_M, measured on the same host and build:** +2.1 / +1.6 / +2.6 %
|
||
(iGPU-only pp512 / pp16384 / tg64), +2.7 % tg32@64k, +4.1 % and +2.0 % for the
|
||
two layer-split configs, ~0 on both pp512@64k. Host RAM bottomed at **10.0 GiB
|
||
available** in A\* against ~5 GiB for Q4_K_M. A memory lever with a small speed
|
||
dividend — nothing structural moves.
|
||
- **IQ4_XS pulled in parallel with the bench**, as asked: 97,473,155,360 bytes,
|
||
sha256 `b4787265…d0574f9d`, matching the Hub's LFS digest. Kept as the fresh
|
||
payload for the next comparison; Q4_K_S was cleared once the bench was done.
|
||
- **Held back deliberately:** `models/checkpoints` (23 GB of Flux 2 Klein — no
|
||
unit mounts it, but its upstream is licence-gated, so it is not mine to
|
||
destroy) plus the small tool dirs under `models/` and `models/logs`, which is
|
||
the evidence for this page.
|
||
- **Verified after the refresh:** `/v1/models` correct on 7770/7771, real
|
||
completions returning `pong` on both, halogen's ledger `68.0 GiB weights
|
||
locked / 11.0 GiB KV pool`, 2 slots over one 400384-position pool, and **307**
|
||
contiguous 2 MiB blocks left (240 before the refresh).
|
||
- Runbook: [[halogen-flash-server]] gains a "Refreshing the weights" note; the
|
||
quant comparison is appended to [[x4-pair-experiment]].
|
||
|
||
## 2026-09-17 — frank-small swapped to Gemma4-12B QAT (Q4_K_M)
|
||
|
||
- **Cutover:** `frank-small` (Dankclaw RTX 3090, :7771) replaced
|
||
ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with
|
||
`HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` Q4_K_M —
|
||
7.38 GiB main + 175 MiB BF16 mmproj (with audio encoder) + 254 MiB
|
||
`mtp-gemma-4-12B-it.gguf` assistant sidecar, all sha256-verified.
|
||
- **Spec decode:** the MTP head is now a separate `gemma4-assistant`
|
||
file loaded via `--model-draft` (shares the target's last layers' KV,
|
||
`llama-model.cpp:2662`), chained behind draftless `ngram-mod`:
|
||
`--spec-type draft-mtp,ngram-mod`. Measured acceptance 0.38–0.46,
|
||
mean draft len ~2.2.
|
||
- **KV:** q8_0 K and V; `build-cuda` rebuilt with
|
||
`-DGGML_CUDA_FA_ALL_QUANTS=ON` (required, else ~12x prefill collapse).
|
||
`CTX_SIZE` unset → `--fit on` chose the full 262,144-token window;
|
||
idle card usage 11,996 / 24,576 MiB (was 22,362).
|
||
- **Template behavior:** Gemma4 accepts every `reasoning_effort` level
|
||
(all seven probed: 200) — the Qwen3.8 gate is gone; harness map kept
|
||
as a safety net.
|
||
- **Verified live:** health 200, chat + `reasoning_content`, `tool_calls`,
|
||
vision (live JPEG), deploy e2e gate passed (31 passed, 1 flaky crawl).
|
||
- **Harness:** `models.yml` frank-small `contextWindow: 262144`.
|
||
- **Note:** the boot log's `Gemma4Assistant requires ctx_other to be
|
||
set` error is `--fit`'s own probe — normal, self-recovers seconds later.
|
||
- Runbook: [[frank-small-gemma4-12b]] (active);
|
||
[[frank-small-qwen38-gsq-rco]] retired 2026-09-17 as rollback target.
|
||
|
||
## 2026-09-17 (later) — frank-small: four full windows, max-length throughput sweep
|
||
|
||
- **Slot ceiling measured:** 1 full-ctx slot idles 11,996 MiB, 4 idle
|
||
22,346 MiB -> 3,450 MiB/slot, ~8,546 fixed; a 5th projects past the
|
||
24,576 MiB card. Deployed `CTX_SIZE=1048576` / `PARALLEL=4` (each
|
||
slot the FULL 262,144 window; explicit CTX bypasses `--fit` on
|
||
purpose — fit would divide the pool at parallel>1). The bats seam now
|
||
pins the division and the budget inequality.
|
||
- **Audio verified:** JFK WAV via `input_audio` -> verbatim transcript.
|
||
- **Batch sweep at max length** (salted API probes, pp ~254k /
|
||
tg @ ~253k ctx): ubatch 512 -> pp 912–924, tg 90–204;
|
||
**ubatch 1024 -> pp 962–965, tg 226–302 (deployed)**;
|
||
ubatch 2048 -> OOM in the MTP draft's pp compute buffer (~3.3 GiB
|
||
vs 2.2 GiB headroom).
|
||
- **Negative results kept:** draft n-max 4 + poll 100 decoded WORSE at
|
||
250k (230–257) — longer verification rounds re-read more long-ctx
|
||
attention than the accepted token earns. No CUDA-graph flags exist in
|
||
this build, so launch-overhead tuning is off the table.
|
||
- **Concurrency caveat:** a 254k prefill beside a decode drops it
|
||
~240 -> ~100–135 t/s; one engine, no per-slot QoS.
|
||
- **Housekeeping during the sweep:** the earlier live service was
|
||
discovered to be an orphaned manual boot holding 7771 past the unit
|
||
stop (kill -9'd); the sweep now refuses to measure a port it does not
|
||
own. Peer frank-medium WIP was stash-held around the deploy and
|
||
self-committed as `23f0978`/`74dda30` — nothing lost.
|
||
- **Live after deploy:** `n_slots = 4, n_ctx_slot = 262144`, idle
|
||
23,300 MiB, e2e gate 32 passed.
|
||
|
||
## 2026-09-17 (late) — homepage: live model info + user docs; deploy-runner lockout fixed
|
||
|
||
- **Reported "www unreachable internally" — server side healthy end to end.**
|
||
LAN DNS → 192.168.0.143, TCP 443 open, Anubis challenge served, `@home`
|
||
200 from inside Caddy, apex/www redirects correct, no 5xx in the journal.
|
||
Fault pattern matches the known client-side split-DNS issue (client off
|
||
Pi-hole → WAN IP, no 443 forward). `frank-large`'s 5 s health timeout
|
||
during the audit was the documented halogen PING stall — 200/1 s minutes
|
||
later, self-recovered; no restart performed.
|
||
- **Fleet sweep:** dankclaw 30 containers up (all with enabled quadlet units
|
||
except `systemd-hermes`, a running container with no quadlet — logged as a
|
||
boot-persistence gap); dankserver media stack + frank-medium all up,
|
||
units enabled; FrankBook-Pro *arr/qB ports all answering (302/200).
|
||
Undeclared stopped container `test-vllm` on dankserver — stray leftover,
|
||
held for approval, not removed.
|
||
- **Security audit:** every vhost classified against the guard policy.
|
||
Guarded (Anubis→Authentik): www, search, grafana, prometheus, loki,
|
||
seerr, marreta, romm, files. Documented exemptions verified
|
||
behaviorally: books anon `/all` → 302 login (gated), ai lands on sign-in,
|
||
WAN 443 still unforwarded from LAN peers, and all admin-only names
|
||
(prowlarr/lidarr/readarr/qb/pihole/hermes/hermesadmin) NXDOMAIN on
|
||
public resolvers. No new exposure found; nothing to secure further.
|
||
- **Homepage shipped (`32751c0`):** the three LLM cards moved to the user
|
||
page with live descriptions (halogen w4b iGPU / 35:30 layer-split MTP
|
||
27B / Gemma4-12B QAT vision path) and `docs: /docs#models`; new static
|
||
user docs page at `/docs` (access chain, model guide, media how-tos,
|
||
troubleshooting), routed in the post-auth listener, shipped by
|
||
`deploy_homepage`, covered by new pages-seam tests.
|
||
- **Runner lockout fixed (`02a7876`+):** a stale `pending` state entry for
|
||
the retired `frank-default-sightsound` endpoint made every `--pending`
|
||
run exit 2 at the dispatcher before the e2e gate. The pending pass now
|
||
prunes entries the target no longer allows into `~/.AGENTTRASH` and
|
||
keeps going; behavioral seam test added, superseded scoping test migrated.
|
||
- **Deploy verified live:** sightsound pruned (state 0, trashed 20260917),
|
||
`/docs` 200 on the post-auth listener, all three model cards `up`,
|
||
bats 329 green, e2e gate 32 passed.
|
||
|
||
## 2026-09-18 (morning) — frank-medium found stopped after a bench round
|
||
|
||
- **Symptom:** connection refused on `192.168.0.235:7770`. Not network —
|
||
the unit itself was `inactive` (enabled, unmasked, `NRestarts=0`).
|
||
- **Root cause (from sudo journal):** the AGENTS.md bench protocol was
|
||
executed from `PWD=/home/dank` — `stop` + `mask` at 02:47:00, `unmask`
|
||
+ `start` at 04:21:27, then a final `stop` at 04:25:59 that was never
|
||
followed by the protocol's step 3 (unmask/start + health verify).
|
||
A bench round ended and left the endpoint down; no crash involved.
|
||
- **Restore:** GPU confirmed idle first (both R9700s ~60 MiB, no
|
||
`llama-bench`/`llama-server` procs), then `systemctl start
|
||
frank-medium`. Health 200 on first probe; `/proc/MainPID/cmdline`
|
||
confirms the deployed geometry (`--tensor-split 35,30`, ctx 393216,
|
||
2 slots, batch 6144x512, `draft-mtp`); a real completion returns
|
||
verbatim. Fingerprint `b11009-fb27a525d` = the pinned runtime.
|
||
- **Lesson:** the restore half of the bench protocol is not optional.
|
||
If you bench from an agent session, the session owes the fleet a
|
||
`health 200` before it ends. Consider making the bench harness
|
||
self-restoring.
|
||
|
||
## 2026-09-18 (midday) — frank-medium down: orphaned merge-build server held both R9700s
|
||
|
||
- **Symptom:** connection refused on `192.168.0.235:7770`. Unit was
|
||
`inactive`, `enabled`, unmasked (`/etc/systemd/system/frank-medium.service`
|
||
symlinked to the repo unit, not `/dev/null`), `NRestarts=0` — a clean
|
||
stop, not a crash.
|
||
- **Not the morning's case:** the GPUs were **not** idle. `rocm-smi` read
|
||
15.5 GiB on GPU0 and 17.1 GiB on GPU1. `ps` named the holder: PID
|
||
2519965, `llama-server` from `third_party/llamacpp/build-merge`
|
||
(gitignored scratch), orphaned at PPID 1, bound `127.0.0.1:17770`,
|
||
`-c 16384 --split-mode layer --tensor-split 1,1 -ngl 99 -t 16`.
|
||
- **Timeline (sudo journal + `/tmp/layersplit.log`):** `start` 08:14:02 →
|
||
`stop` 09:19:44 → merge build spawned 09:19:47 → log reached
|
||
`model loaded` / `listening` at 09:20 → nothing after. The merge-build
|
||
load validation passed; the cleanup-and-restore half never ran.
|
||
- **Why prod could not come back:** the 35:30 split needs ~30.8 GiB on
|
||
ROCm0 and ~28.9 on ROCm1. The stray owned roughly half of each card, so
|
||
`systemctl start` would have died on allocation, not on config.
|
||
- **Proof it was abandoned, not parked:** `/proc/<pid>/stat` utime+stime
|
||
identical across a 10 s sample (frozen CPU), zero established
|
||
connections to `:17770`, loopback-only bind, 3h05m since last log write.
|
||
- **Restore:** preserved the test log to `/tmp/layersplit.log.preserved-20260918`,
|
||
`kill 2519965` (exited in <1 s, both cards → 60 MiB baseline), then
|
||
`systemctl start frank-medium`. Health 503 → 200 in ~20 s (payload warm
|
||
in page cache). `/proc/MainPID/cmdline` confirms deployed geometry
|
||
(`--tensor-split 35,30`, ctx 393216, 2 slots, `--flash-attn on`,
|
||
`--spec-type draft-mtp`); steady state ROCm0 32.9 GiB / ROCm1 30.7 GiB.
|
||
Real completion returns verbatim with `reasoning_content` and
|
||
spec-decode engaged: `draft_n 39 / accepted 35` (89.7 %), tg 40.6 t/s.
|
||
Fingerprint `b11009-fb27a525d` = the pinned runtime.
|
||
- **Lesson:** "GPUs idle" is a check, not an assumption. A stopped prod
|
||
unit plus busy cards means something else took them — find the holder with
|
||
`ps`/`ss` and prove it abandoned (frozen CPU ticks, no conns) before
|
||
reclaiming. Scratch servers on loopback are invisible to the endpoint
|
||
monitor, so they strand the fleet instead of failing loudly.
|
||
|
||
## 2026-09-18 (afternoon) — frank-medium: the host-RAM prompt cache doubles to 16 GiB
|
||
|
||
- **The old cap was binding, not theoretical.** `--cache-ram` caps the total
|
||
bytes of saved prompt states in HOST RAM (`server_prompt_cache`), and
|
||
`alloc()` refuses any single state larger than the whole cap with one `WRN`.
|
||
The journal held **137** such refusals in the two days from 2026-09-16 —
|
||
`prompt state size 11388.990 MiB exceeds cache size limit 8192.000 MiB,
|
||
skipping` — i.e. every conversation past ~170k tokens was uncacheable, and
|
||
its next turn re-prefilled the whole thing.
|
||
- **The cap is a token budget in disguise.** A state is the serialized KV of a
|
||
whole sequence plus its constant-size DeltaNet state plus the MTP draft
|
||
context's own KV: 16 full-attention layers x 4 kv heads x 256 dim x 2 B x 2
|
||
(K,V) = 64 KiB per token, plus 4096 B for the draft context. So 8192 MiB held
|
||
~120k tokens and 16384 MiB holds ~240k — and a FULL 196608-token slot state
|
||
(~12.8 GiB), which the old cap refused outright, now fits with ~3 GiB spare.
|
||
- **Deployed change** (`1acca43`): `CACHE_RAM` 8192 → 16384 in
|
||
`scripts/frank-medium.env`, with the law and the journal evidence in the
|
||
file. Host RAM, allocated on demand and never pinned, so no VRAM number
|
||
moves; bounded rather than `-1` because dankserver shares its 62 GiB with the
|
||
media-stack co-tenants.
|
||
- **New seam** (`tests/frank_medium_unit.bats`): the cap must clear one full
|
||
slot's state and may not claim more than half the host's RAM. It fails on the
|
||
old value (8192 < 13056 MiB) and passes on the new one — checked both ways.
|
||
- **Adjacent correction:** the same block claimed 0 was the build's default for
|
||
`--ctx-checkpoints`; this binary's `--help` says **32**, so the 0 is a
|
||
deliberate deviation. Fixed in the env comment and in the runbook's flag
|
||
table.
|
||
|
||
## 2026-09-20 — frank-medium cut over to radiance TP=2; llama layer-split retired
|
||
- **Measurement.** Three arms, one salted battery (`scripts/bench-conversation.py`,
|
||
10 prompts, streaming, TTFT + decode tok/s, `--burst N`), the service stopped
|
||
and masked per the benchmarking policy for every arm. radiance TP=2: decode
|
||
geomean 95.8 t/s, TTFT 0.21 s, prefill geomean 1493.6 t/s, burst-4 aggregate
|
||
259.4 t/s (65.6 per stream). llama.cpp `--split-mode tensor` 1:1: 30.5 t/s.
|
||
llama.cpp layer split 32:33 (the incumbent): 45.7 t/s, 0.72 s, 71.7 t/s burst.
|
||
Radiance won every axis; commit `c211665` cut the endpoint over.
|
||
- **The pre-probe came out of the harness first.** A warm prefix-cache hit hid
|
||
prefill, so the first row of every arm had been measuring the cache, not the
|
||
engine.
|
||
- **llama.cpp tensor mode cannot carry the drafter on this build.** `--split-mode
|
||
tensor` plus the DFlash2 drafter aborts at `ggml-backend.cpp:941` (a weight
|
||
pre-allocated on a non-primary device has no backend when `n_copies == 1`), so
|
||
arm L ran with NO speculative decoding — its 30.5 t/s is therefore not a
|
||
like-for-like against radiance's spec-decoded 95.8, and is reported that way in
|
||
[[Dual-R9700-Cooperation]]. `--split-mode row` is SYCL-only (it needs
|
||
`ggml_backend_split_buffer_type`); `tensor` is the HIP/CUDA mode.
|
||
- **A collective HAS been dispatched on this board since the swap**, which
|
||
supersedes the "unblocked, not proven" line in the entry above.
|
||
`scripts/build-llama-rdna4.sh` now takes `GGML_HIP_RCCL=${GGML_HIP_RCCL:-OFF}`
|
||
and bundles `librccl.so*` + `librocm-core` when ON; the ON build
|
||
(`build-rdna4-rocm714-rccl`) is what ran arm L. Production stays OFF. Ubuntu
|
||
RCCL 2.27 carries gfx1201 device code; TheRock's does not ("invalid device
|
||
function"), which is why radiance ships the collective-free R4D one-shot
|
||
allreduce instead of depending on RCCL at all.
|
||
- **Deploy.** `deploy_llm_medium` in `scripts/deploy-remote.sh` gates on image
|
||
tag, MXFP4 checkpoint, drafter, patched `libr4d`, and the chat template; stops
|
||
and trashes the root llama unit (the relinked symlink is MOVED, never
|
||
`systemctl disable`, which would delete the file); waits for 7770 to be
|
||
released; installs the quadlet; waits for container health. Deployed healthy in
|
||
181 s at `max_model_len` 262144, TP=2, 8 conversations, DFlash2 depth 7.
|
||
- **The KV pool ships auto-sized.** The unit sets no `KV_CACHE_MEMORY_MB`, so the
|
||
engine sized the pool itself: 943,581 tokens = 3.59 concurrent full 262,144-token
|
||
conversations. Under a 4-way burst the container cgroup stayed flat at 28.4 GiB
|
||
RAM and 3.3 GiB swap with host MemAvailable ~48 GiB, so the 2026-09-16 swap
|
||
storm (measured on TWO vision-carrying replicas at ~943k each) does not
|
||
reproduce on this shape. Arm R ran the explicit 18563072000 pin — 262,720
|
||
tokens, 1.00x full window — and production prefers the 3.59x.
|
||
- **Capabilities verified live rather than assumed.** Tool calls return
|
||
`tool_calls` with `finish_reason: tool_calls` under
|
||
`--tool-call-parser qwen3_coder`. `reasoning_effort` low/medium/high/none all
|
||
answer 200, and both `none` and `chat_template_kwargs.enable_thinking=false`
|
||
switch thinking off, so this endpoint needs no `reasoningEffortMap` (that
|
||
workaround belongs to Qwen's restricted template on the Dankclaw endpoints).
|
||
Thinking arrives as vLLM's `reasoning`, not llama.cpp's `reasoning_content`.
|
||
**Vision works**: the loader installs the Gemma4 vision path and the endpoint
|
||
answered two held-out synthetic images correctly ("This is a red circle." /
|
||
"This is a blue rectangle."), so the `language_model_only` → text-only premise
|
||
written earlier today was wrong and `models.yml` keeps `input: [text, image]`.
|
||
- **Production reproduces the winning arm**: same battery, 92.4 t/s decode
|
||
geomean, 0.22 s TTFT, 1213.3 t/s prefill, 248.9 t/s burst-4 — arm R within
|
||
noise.
|
||
- **Harness.** `~/.omp/agent/models.yml` frank-medium now declares
|
||
`contextWindow: 262144` (the pool is 3.59x the window and preempts rather than
|
||
refusing, so the window is the honest number), `maxTokens: 32768` inside the
|
||
compaction gap, `input: [text, image]`, no effort map. A
|
||
`omp -p --no-session --model frank-medium` tool round trip returned the
|
||
command's output.
|
||
- **Monitoring cutover.** `shared/prometheus/rules/llm-fleet.yml` gained the vllm:
|
||
half — `LlmRequestsWaiting`, `LlmKvCacheSaturated`,
|
||
`job:llm_vllm_prefix_cache_reuse:rate5m`, `job:llm_spec_accept_length:rate5m` —
|
||
and the two existing collapse/spec alerts now cover both namespaces with a
|
||
traffic floor on each side. Every expression was run against the live
|
||
Prometheus; the accept-length rule answers 4.09 for frank-medium right now.
|
||
`shared/prometheus/prometheus.yml` also fixed a target that was red by
|
||
construction: from this bridge-networked Prometheus a host-process bind is
|
||
reachable at `host.containers.internal`, NOT at the LAN IP (netavark DNATs
|
||
published ports only), so `frank-small` had read down for days while healthy
|
||
and exporting. `frank-large` is now scraped too — it exports 19 `llamacpp:`
|
||
families including every series the rules use, contra the comment that excluded
|
||
it.
|
||
- **The two-NIC question from the entry above is decided**: `eno1` is pinned as
|
||
the primary route (metric 100) with `eno2` retained as failover (metric 200) by
|
||
`scripts/dankserver-route-primary.sh`, applied with `nmcli device reapply`
|
||
(`nmcli connection reapply` does not exist in NetworkManager 1.54).
|
||
- **Rollback.** `git revert c211665`, relink `scripts/frank-medium.service` from
|
||
`scripts/.AGENTTRASH/20260920-llama-layersplit/`, `sudo systemctl restart
|
||
frank-medium.service`. The llama runtime and the Q8_0 payload are still on disk.
|