Frank
17450dd172
- runbook: identity now 4x262144, audio verified, throughput table with the OOM and negative-result rows, deployed flags updated to reality. - AGENTS.md frank-small paragraph matches the deployed shape. - log.md: the sweep entry. - scripts/frank-small.env: my earlier line-range edit had clobbered NGL=999 and FLASH_ATTN=on out of the file; the seams caught it. The live service was unaffected because the script's defaults match the declared values exactly, but the file contract is restored. Suite: ALL TESTS PASSED (321), vault 125 pages 0 unresolved.
851 lines
66 KiB
Markdown
851 lines
66 KiB
Markdown
---
|
||
type: reference
|
||
title: Bundle Change Log
|
||
description: Chronological record of meaningful changes to the serverAppConfigs knowledge base.
|
||
tags:
|
||
- changelog
|
||
- documentation
|
||
- maintenance
|
||
timestamp: 2026-09-14
|
||
---
|
||
|
||
# Bundle Change Log
|
||
|
||
## 2026-09-16 (late) — frank-medium: ubatch's optimum is 512, and it moved the split to 35:30
|
||
- **New harness, shipped before use** (`scripts/bench-batch-ubatch.sh`, `87845fc`,
|
||
fixed in `77a07ff`): one `llama-bench` invocation per cell — a test that cannot
|
||
allocate calls `exit(1)`, so one infeasible ubatch would take the whole sweep
|
||
with it — a VRAM sampler per cell, and the service stopped for the run and
|
||
started again from a trap. It must run from a transient unit: the deploy's
|
||
`LimitMEMLOCK=infinity` is load-bearing for ROCm staging buffers and an SSH
|
||
shell on these hosts only gets 8 MiB.
|
||
- **The sweep** (23 of 28 cells, a 64k prefill measured *at* 64k depth, 3 reps):
|
||
prefill **peaks at ubatch 512**. 256 and 384 lose ~11 % (SMs not saturated),
|
||
2048 loses 10 %, and 768–1792 sit in a shallow basin 3–5 % below the peak. The
|
||
curve is deterministic rather than noisy — 512 measured 1192.88 / 1193.23 /
|
||
1193.28 / 1193.88 across batch 2048 / 4096 / 6144 / 8192, and ubatch 2048
|
||
measured 1077.38 / 1077.54 / 1077.59 — which is what makes the 2.4 % gap
|
||
between 512 and the deployed 1536 real.
|
||
- **The logical `--batch-size` is irrelevant**: 2048 through 8192 land within 2 %
|
||
of each other at a fixed ubatch and use identical VRAM, because the compute
|
||
buffers are sized by ubatch alone. Decode never moves at all — 15.88 tok/s at
|
||
64k depth on every cell, 17.80 at depth 0 — while depth itself costs a uniform
|
||
~32 % of prefill.
|
||
- **Deployed change** (`a11fae0`): `UBATCH_SIZE` 1536 → 512, which is both faster
|
||
and 3.71 GiB cheaper, and `TENSOR_SPLIT` 36,29 → 35,30, because the ubatch
|
||
buffers cost ROCm1 nearly twice what they cost ROCm0 (1.30 against 2.41 GiB
|
||
for the 1024-unit move, measured at 262144 over two slots) and shrinking them
|
||
handed ROCm1 1.31 GiB more than ROCm0. Live after the deploy: **6.44 / 6.90
|
||
GiB free** (was 4.71 / 4.91), 6.23 / 6.64 under two concurrent requests, and a
|
||
0.46 GiB gap — one DeltaNet layer, which is the granularity floor here.
|
||
- The balance seam in `tests/frank_medium_unit.bats` was re-fitted at the new
|
||
deployed point with both ubatch coefficients measured at this shape; every
|
||
neighbouring split still fails it (36,29 → 1.85 GiB, 34,31 → 0.94, 37,28 →
|
||
3.24, 1,1 → 3.03, 39,26 → 6.03). The unit test's literal `TENSOR_SPLIT=36,29`
|
||
assertion became a *shape* assertion — two shares summing to the model's 65
|
||
blocks with ROCm1 taking the smaller — so re-measuring the balance no longer
|
||
breaks a test about parallelism.
|
||
- **Measurement trap worth keeping**: re-loading immediately after a kill fails
|
||
with `unable to allocate ROCm0 buffer`. amdgpu does not hand ~26 GiB back at
|
||
once, and a 180 s settle is not enough; it reads exactly like a configuration
|
||
that does not fit. The measurement helper grew a wait-for-empty gate.
|
||
- Docs: [[frank-medium-batch-ubatch]] (new page), the runbook's deployed flags
|
||
and split history, and AGENTS.md's batch lesson. [[Dual-R9700-Cooperation]]
|
||
gained the live `AtomicOpsCap` / `AtomicOpsCtl` chain — `ReqEn+` on card 0
|
||
against `ReqEn−` on card 1, because the kernel refuses the requester bit when
|
||
a bridge cannot route atomics — and its M.2_2 riser premise is now labelled a
|
||
vendor claim rather than a fact.
|
||
|
||
## 2026-09-15 — frank-medium: Dankserver GPU swap (2x R9700), frank-small retired
|
||
- The RTX 3090 left Dankserver; two AMD Radeon AI PRO R9700 cards (Navi 48,
|
||
gfx1201, 32,624 MiB each) replaced it. FreeToken/CUDA cannot run there, so
|
||
`frank-small` is retired and `frank-medium` took the :7770 endpoint.
|
||
- Runtime: charlie12345/ROCmFPX llama.cpp `c49ebdb` built for **gfx1201** with
|
||
the fork's self-contained ROCm 7.14 toolchain (TheRock nightly, RPATH-bundled
|
||
libs) — the host's broken mixed ROCm (AMD-repo 6.3.3 hijacking libhsa via
|
||
`/etc/ld.so.conf.d/10-rocm-opencl.conf`) is fully bypassed. Model:
|
||
kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF (ftype 115, 26.28 GiB) +
|
||
Q4_0 MTP draft head; mmproj unused (vision stays on frank-large).
|
||
- **Parallelism decision, measured not argued:** llama-bench with the correct
|
||
multi-device syntax (`-dev ROCm0/ROCm1` — comma form silently benches each
|
||
card separately): layer 1991/18.8 t/s (pp4096/tg128) vs tensor 271.8/26.3 vs
|
||
single card 1160.9/19.4. Tensor's decode win costs a 7x prefill collapse
|
||
(no RCCL → "meta-backend butterfly" allreduce); served with MTP, layer also
|
||
wins decode (35.7–46.2 tok/s vs 28.7). Operator picked layer ("pipeline
|
||
parallelism"); the data agrees on both axes. Draft head pinned to ROCm1
|
||
(card holding the last target layers); under tensor mode an unpinned draft
|
||
aborts the KV build (GGML_ASSERT(buffer) in ggml-backend.cpp).
|
||
- Verified live before and after the systemd cutover: health ~18 s, MTP
|
||
acceptance 60–69 %, cold 949-token prefill in 2036 ms, and **prefix caching
|
||
works with the draft head** (turn 2 reprocessed 22 tokens, cache_n 945) —
|
||
the model card's mtp-prompt-cache-fix.patch defect is already fixed in
|
||
`c49ebdb` (`common_prompt_checkpoint::data_spec`).
|
||
- Deploy: `deploy.sh dankserver --service frank-medium` (new
|
||
`retire_root_llm_small()` archived the CUDA-era unit to
|
||
`~/.AGENTTRASH/20260915-frank-small-retire/host-units/`); e2e gate 30/30
|
||
including 11-admin (the blankpad→192.168.0.112 path issue has cleared).
|
||
- Host cleanup (the "Both" half of the plan): purged 167 stale AMD-repo
|
||
`~24.04` ROCm 6.3.3 packages, removed the radeon apt pin and the
|
||
ld.so.conf hijack (staged in `~/.AGENTTRASH/20260915-rocm633-cleanup/`);
|
||
`libhsa-runtime64` now resolves only to Ubuntu's ROCm 7.1 and `rocm-smi`
|
||
7.1.1 (Ubuntu package) works. frank-medium unaffected (self-contained).
|
||
- Client cutover: blankpad `models.yml`/`config.yml` renamed the model id to
|
||
`frank-medium` (ctx 131072, maxTokens 32768); Hermes live config on Dankclaw
|
||
repointed `dankserver-small` routes to `frank-medium`.
|
||
- New: `/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md` runbook;
|
||
updated AGENTS.md, README, docs indexes, Dankserver System Specs,
|
||
deployment/Hermes/OpenWebUI/Plex/Stack pages; freetoken-frank-small and
|
||
kingjones-q8agent marked retired/superseded.
|
||
|
||
## 2026-09-12 — Marreta content-cleaning proxy deployed on Dankclaw
|
||
- Added `manualdousuario/marreta` (Laravel / `serversideup/php:8.4-fpm-nginx`)
|
||
as a rootless Quadlet pair: `marreta` + `marreta-browser` (Lightpanda),
|
||
both joining **gluetun's netns** so all fetch egress exits the same
|
||
ProtonVPN tunnel as [[SearXNG]].
|
||
- **Port-collision fix:** the shared netns already has SearXNG on 8080, so
|
||
marreta is relocated to `NGINX_HTTP_PORT=8085` (verified against the
|
||
serversideup source: the image does not bake `conf.d/*.conf`; the
|
||
entrypoint `envsubst`s `${NGINX_HTTP_PORT}` at start). Gluetun publishes
|
||
`8082:8085` on marreta's behalf; Caddy dials `192.168.0.143:8082`.
|
||
- **Egress rationale:** netns sharing, not `PROXY_URL` — the browser's CDP
|
||
fetch passes no proxy, so only netns keeps the browser off the home IP.
|
||
- **Guard chain:** `marreta.franksdank.cloud` → Anubis `:8923` → Authentik
|
||
→ post-auth `:8017` `@marreta` → `192.168.0.143:8082`. Access model is
|
||
**any authenticated user** (no group gate). Requires a one-time
|
||
`ProxyProvider` (external_host `https://marreta.franksdank.cloud/`,
|
||
internal_host `http://caddy:8017`) on the embedded outpost.
|
||
- Secrets seeded to `/home/dank/Containers/marreta/env` (0600, dank-owned)
|
||
from `MARRETA_APP_KEY` / `MARRETA_ADMIN_EMAIL` / `MARRETA_ADMIN_PASSWORD`.
|
||
- New test seam `tests/marreta_quadlet.bats` (17 contracts). deploy-plan.py:
|
||
marreta joins the gluetun netns family. Doc: [[Marreta]].
|
||
- **Deployed live 2026-09-12 (rev `91e9410`)** after standing up passwordless
|
||
SSH from blankpad to the fleet. Ran `deploy.sh dankclaw --service gluetun
|
||
--service marreta --service caddy` (gluetun first: the new `8082:8085`
|
||
publish needs a tunnel recreate, which also cycles SearXNG). Authentik
|
||
`ProxyProvider` `marreta` registered (pk 13). Verified: unauth → Anubis
|
||
challenge; post-auth `:8017` → 200; container healthy; netns sharing proven;
|
||
SearXNG unaffected.
|
||
- **Gotcha fixed in-flight:** the first seed used a bare-base64 `APP_KEY`,
|
||
which 500s every Laravel app route (`Unsupported cipher or incorrect key
|
||
length`). Re-keyed with the required `base64:` prefix before any encrypted
|
||
data persisted. See [[Marreta]] § Secrets.
|
||
- **Login-wall fixed 2026-09-12 (post-deploy).** The homepage Marreta link
|
||
parked on `marreta.franksdank.cloud/if/flow/default-authentication-flow/`
|
||
instead of the app. Root cause chain, in the order found:
|
||
1. `marreta-proxy` had `invalidation_flow = null`, which crashed the Rust
|
||
embedded outpost's serde (`invalid type: null, expected a formatted UUID
|
||
string`) and took down **every** provider on the outpost. Set to
|
||
`default-provider-invalidation-flow`.
|
||
2. `authentication_flow` / `authorization_flow` were null; set to
|
||
`default-authentication-flow` / `default-provider-authorization-implicit-consent`.
|
||
3. **The real blocker:** the outpost's config API filters
|
||
`ProxyProvider.objects.filter(application__isnull=False)`. `marreta-proxy`
|
||
had no linked **Application**, so the outpost never loaded it and the
|
||
request fell through to raw flow serving on the marreta host — where the
|
||
IdP session cookie (scoped to `auth.franksdank.cloud`) is invisible.
|
||
Created `Application(name="Marreta", slug="marreta", provider=marreta-proxy)`.
|
||
- After `systemctl --user restart authentik-server authentik-outpost`, the
|
||
outpost loaded `marreta-proxy` and the OAuth dance completes: `/` →
|
||
`/outpost.goauthentik.io/start` → `auth…/o/authorize` → callback → Marreta.
|
||
Regression test: `tests/e2e/specs/16-marreta.spec.ts`. Providers remain
|
||
**live-only (no repo blueprint)** — a provenance gap to close.
|
||
|
||
## 2026-09-04 — *arr fleet consolidated on FrankBook-Pro; all egress VPN-gated
|
||
- Operator decision: **FrankBook-Pro is the sole *arr deployment.** The
|
||
Dankserver compose copy (booted earlier the same day) was removed;
|
||
`/opt` config roots trashed on the host; compose/services pruned.
|
||
- All six fleet containers (qB + five arrs) now share gluetun's network
|
||
namespace — **all egress exits the ProtonVPN tunnel by construction**
|
||
(AllowedIPs=0.0.0.0/0). The arrs publish no ports of their own; their UI
|
||
ports (9696/8989/7878/8686/8787) moved onto the gluetun unit. Proven live:
|
||
residential `71.227.227.172` vs netns-wide `51.179.177.14`.
|
||
- New test seam `tests/frankbook_quadlet.bats`: every arr+qB unit must join
|
||
gluetun's netns and publish no port; gluetun must publish the whole
|
||
fleet's port set. deploy-plan.py: gluetun changes carry the whole netns
|
||
family.
|
||
- arr→qB download clients re-pointed `10.88.0.1:8080` (old bridge-gateway
|
||
hack, dead inside the tunnel) → `127.0.0.1:8080`; verified by real qB
|
||
login from inside the netns. Caddy vhosts + homepage probes repoint to
|
||
192.168.0.112; homepage all green.
|
||
- Defects discovered: Prowlarr's four Application rows are GONE (empty
|
||
table; sync broken until rebuilt with loopback baseUrls) and readarr's
|
||
qB proxy auth defect needs a retest post-move. EZTV/1337x CF-block
|
||
parking may be stale now that egress is VPN — retest before enabling.
|
||
- E2E specs (11-admin/12-crawl) de-hardcoded: expected open-link counts now
|
||
derive from the live services.json registry. AGENTS.md: deployments are
|
||
complete only when unit tests AND Playwright E2E both pass.
|
||
|
||
## 2026-09-04 — dankserver *arr stack reactivated; rootless identity model documented
|
||
- ARR-Stack: the Dankserver compose deployment (committed 2026-08-29, never
|
||
booted — root-owned `/opt/<svc>/config` dirs crashed every app at logger
|
||
init before any port bound) is live as of 2026-09-04 behind six own-auth
|
||
TLS vhosts (`<svc>.franksdank.cloud`), all verified 200, homepage probes
|
||
green. Documented the verified rootless identity model — container root
|
||
(PUID=0) maps to host uid 1000 (dank) by default on podman 5.7.0
|
||
(`/proc/<pid>/uid_map` = `0 1000 1`, `1 100000 65536`), so data roots are
|
||
`1000:1000` and **`userns_mode` must never be added to compose services**
|
||
(podman-compose 1.5.0 runs services in pods; `--userns` + `--pod` is
|
||
rejected). FrankBook-Pro differences noted: no gluetun egress gate on the
|
||
Dankserver qB, empty (fresh) app configs, Sonarr in the homepage user
|
||
section and the rest in admin.
|
||
## 2026-09-03 — full fact-check sweep of the docs tree (docs-only, deploy no-op)
|
||
- Scope: every page under `/serverAppConfigs-docs/` re-verified against live
|
||
hosts (read-only SSH), quadlet unit files, and the Caddyfile; surgical edits
|
||
only — no pages created, renamed, or deleted. Per the deploy planner, a
|
||
docs-only change is a deploy no-op.
|
||
- Dankserver core rewritten to live quadlet facts: the *arr stack + qBittorrent
|
||
+ gluetun run on **FrankBook-Pro** (units installed to
|
||
`/home/core/.config/containers/systemd/` via the frankbook-pro deploy
|
||
handler; `PUID=0/PGID=0`, data roots `/Volumes/COMPLETE/<svc>-data:/config`;
|
||
qB pinned `4.6.7` sharing gluetun's netns with 8080 + 6881 published on the
|
||
gluetun unit; readarr pinned `amd64-1.0.980`; no vhosts — direct IP:port,
|
||
exempt by construction from the guard chain). Jellyfin, Plex, RomM (8085→8080
|
||
guarded), Calibre-Web (`0.6.27-ls397` + universal-calibre mod; no standalone
|
||
calibre-server), mariadb/redis/Tautulli/Entropy verified against live units.
|
||
- Caddy guard chain documented to the live mechanism end-to-end: Anubis :8923 →
|
||
internal :8016 → post-auth :8017 routing; guarded vhosts (search, grafana,
|
||
prometheus, loki, seerr) route post-auth to container-name upstreams;
|
||
exemptions named by construction reason (auth/IdP itself, gitea smart-HTTP,
|
||
ai→:3005 own auth, hermes/hermesadmin internal-only via DNS absence, pihole
|
||
basicauth edge → :8083).
|
||
- `Gitea.md`: data roots + secrets corrected to the live unit paths
|
||
(`/home/dank/Containers/gitea/{data,postgres,env}`); UID gotcha corrected —
|
||
the units carry no `User=`/UserNS override, so container uid 1000 is literally
|
||
host uid 1000 (no subuid offset for these containers).
|
||
- Dedupe: `Prowlarr.md`/`prowlarr.md` collapsed into one canonical lowercase
|
||
page via rename (git history retains both versions; index link unchanged).
|
||
- LLM docs: archive pages (`DFlash DSpark Speculative Decoding`,
|
||
`ling-3.0-flash-trial`) reworded from current-tense to dated historical
|
||
framing with pointers to the active runbook [[kingjones-strix-rocmfp4]];
|
||
LLM `index.md` + `deployment.md` fleet tables and health-gate text updated to
|
||
the 2026-08-30 fleet state (STRIX on the pinned ROCmFPX fork; Huihui
|
||
abliterated FP8 on FreeToken; sightsound = FLM v1.0.1 on XDNA2, port 7771,
|
||
health `/v1/version` — FLM has no `/health` route).
|
||
- Suite green after the sweep: `./tests/run.sh` → 233 pass / 0 fail, including
|
||
the docs-completeness seam that requires every page to stay index-linked.
|
||
## 2026-09-01 (late) — FrankBook-Pro stack completed: tunnel live, qB wired into the arrs
|
||
- WireGuard credentials provisioned (second Proton config, server `PA-12`,
|
||
endpoint `51.179.177.2:51820`) → `/home/core/Containers/gluetun/env`
|
||
(0600, VM-local); `deploy.sh frankbook-pro --service gluetun --service
|
||
qbittorrent` brought both up; egress verified (Proton PA exit, ~19 MB/s
|
||
on a live torrent). The `.conf` stays in `~/Downloads`; the private key
|
||
never enters the repo.
|
||
- gvproxy binds wildcard `*:6881/udp` (Go sets `SO_REUSEADDR` on TCP but not
|
||
UDP) → the socat 6881 tcp+udp LaunchAgents collided at every boot and
|
||
were retired; swarm inbound (if ever needed) is the Proton NAT-PMP path.
|
||
macOS application firewall still filters gvproxy inbound from non-local
|
||
sources (stealth mode) — cross-host 8080 needs `socketfilterfw` allow.
|
||
- qB wiring: first-seed bug (hash under legacy `WebUI\Password` key, ignored
|
||
by qB ≥ 4.2) fixed live (`WebUI\Password_PBKDF2`) and in
|
||
`scripts/deploy-remote.sh`; `DefaultSavePath` repointed to
|
||
`/Volumes/COMPLETE/Finished Torrents` — qB 5.2 API adds with a category
|
||
do not inherit the category save path (no auto-TMM); download clients
|
||
created in all four arrs (sonarr/radarr/lidarr via API, tests pass;
|
||
readarr by direct `readarr.db` insert — its EOL validator rejects qB 5's
|
||
`204` login as auth failure); Prowlarr Applications sync sonarr/radarr/
|
||
lidarr (readarr omitted: prowlarr's indexer push 500s on EOL schema).
|
||
Legacy `Prowlarr-*` indexer rows (hostname `prowlarr:9696`, unresolvable
|
||
on the Mac default network) deleted. E2E: Sintel fetched + downloaded
|
||
through the tunnel at ~19 MB/s, completed into `Finished Torrents`.
|
||
- Docs: [[FrankBook-Pro]] rewritten to the active-tunnel reality.
|
||
|
||
|
||
## 2026-09-01 - WG endpoint identified as ProtonVPN Seattle; Mac gate is a second dashboard config
|
||
- RDAP: `149.102.254.77` = CDNEXT-CG-SEA (Datacamp Limited, Proton's Seattle
|
||
hosting partner); Dankclaw's tunnel v6 `2a07:b944::2:2` sits in
|
||
CH-PROTONVPN-20160719; observed exit IP `149.22.88.19` = CDNEXT-SEA-CG.
|
||
The "custom WG server" is a ProtonVPN Seattle WireGuard server - there is
|
||
no peer-registration step on any box of ours (full 65535-port scan: no
|
||
SSH anywhere, just silent TLS listeners - 443, 4443-4468, 5995, 7770,
|
||
8443 - Proton's protocol-stealth port map, no admin surface).
|
||
- Mac gate corrected: provision a **second** Proton WG config (dashboard,
|
||
named e.g. `frankbook`) - one config = one keypair; two hosts on
|
||
Dankclaw's config bounce the server-side endpoint and kill SearXNG. Values
|
||
paste straight into the gluetun `custom` env. Earlier prepared keypair on
|
||
dankclaw (`~/frankbook-wg/`) retired to `~/.AGENTTRASH/` (Proton generates
|
||
the keypair per config; no BYO pubkey).
|
||
- [[FrankBook-Pro]] provisioning section rewritten.
|
||
|
||
## 2026-09-01 — Jellyfin authentication moved to Authentik LDAP
|
||
- `/dankclaw/authentik/quadlet/authentik-outpost-ldap.container` (`4ad85fa`,
|
||
digest-pinned in `f4ef2b8`) — new `authentik-outpost-ldap.service` on
|
||
Dankclaw serving LDAP :389/:636 for Jellyfin. Image
|
||
`ghcr.io/goauthentik/ldap` pinned by **amd64 single-platform digest**:
|
||
podman 5.7.0 refuses the tag because it resolves to a nested manifest
|
||
list ("unexpectedly received a manifest list instead of a manifest for a
|
||
single image") — resolve via `podman manifest inspect | jq` and pin.
|
||
- Authentik objects provisioned via `ak shell` ORM (live, idempotent
|
||
get_or_create): LDAP provider `Jellyfin LDAP`, service account
|
||
`svc-jellyfin-ldap` + bind token, groups `jellyfin_users` /
|
||
`jellyfin_admins`, application `jellyfin`.
|
||
- `/scripts/deploy-remote.sh` (`691da47`) — `authentik-outpost-ldap` added
|
||
to `service_is_allowed` (the deploy runner has its own allowlist beyond
|
||
deploy-plan.py's).
|
||
- Two authentik-side rules proven by live bind tests (`ldapsearch` through
|
||
the published port):
|
||
1. Bind token **intent must be `verification`** — `app_password` tokens
|
||
are rejected by the authorization flow's password stage with
|
||
`Invalid password`.
|
||
2. Search requires the **`search_full_directory`** object permission on
|
||
the provider (granted via RBAC role `svc-jellyfin-ldap-search`);
|
||
without it the outpost returns only the bind user, so `memberOf`
|
||
filters match zero entries.
|
||
- Jellyfin side (host-only, no repo change): LDAP-Auth **23.0.0.0**
|
||
installed from the GitHub releases zip (the `repo.jellyfin.org` stable
|
||
manifest path 404s), config XML at
|
||
`config/plugins/configurations/LDAP-Auth.xml`, username attribute `cn`
|
||
(authentik 2026.x maps `uid` to an opaque hash), and — the big one —
|
||
**`LdapSearchAttributes` is comma-separated** (`Split(',')` in the plugin
|
||
source; semicolons silently produce "Found no users matching X").
|
||
End-to-end verified with a throwaway `jellyfin_users` member
|
||
(`AuthenticateByName` 200, non-admin policy), then removed from both
|
||
authentik and the Jellyfin DB directly (its admin-token cleanup 401'd —
|
||
the admin password in the session archive was OCR-corrupted; verified
|
||
offline against the stored PBKDF2-SHA512 hash that the real bootstrap
|
||
value still authenticates, unchanged).
|
||
- Docs: [[Authentik]] gained the LDAP-outpost section (objects table +
|
||
rules), [[Jellyfin]] gained the LDAP-Auth plugin section. Docs-only
|
||
commit → deploy no-op.
|
||
|
||
## 2026-09-01 — frank-small: two compressed-tensors FP8 swaps fail to load; Huihui restored
|
||
- `/scripts/frank-small.env`, `/scripts/serve-frank-small.sh`,
|
||
`/scripts/frank-small.service` (`af6cd72` → `34ab6b4` → reverted
|
||
in `4d5bea4`) — attempted model swaps on FreeToken v0.1.2, both
|
||
live-verified as loader failures and reverted the same day to the Huihui
|
||
FP8 checkpoint:
|
||
- `ornith-ai/Ornith-1.5-35B-A3B-FP8` (compressed-tensors per-channel):
|
||
`ValueError: Missing MoE expert source layers: {'gate_up': ...}` — the
|
||
checkpoint's expert tensors do not follow the
|
||
`mlp.experts.{N}.gate_proj/up_proj/down_proj` naming the MoE provider
|
||
scans for.
|
||
- `lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8`
|
||
(compressed-tensors block-fp8 128×128, standard naming): experts
|
||
resolve but `_dequant_fp8_weight` multiplies `weight × weight_scale`
|
||
directly — block scales do not broadcast →
|
||
`RuntimeError: The size of tensor a (2048) must match ...`.
|
||
- Both failed cleanly at load (37 GB fits RAM — no swap-thrash, no
|
||
outage); service was back on Huihui within the hour and re-verified with
|
||
a real completion (reasoning parses; `17*23 → 391`, natural stop).
|
||
- `/serverAppConfigs-docs/LLM/freetoken-frank-small.md` — quant
|
||
matrix updated: modelopt block-fp8 is the ONLY FP8 path that loads; both
|
||
compressed-tensors shapes documented with their exact errors.
|
||
- **Deploy health-check trap (second occurrence)**: `deploy.sh` reported
|
||
"healthy after approximately 6s" for BOTH failed loads — the health gate
|
||
passes while the doomed worker is still starting. A model swap is only
|
||
proven by the journal (Uvicorn startup) + a real completion.
|
||
- Staged for trash (awaiting purge approval): `/home/dank/models/ornith-1.5-35b-a3b-fp8`
|
||
(37 GB) and `/home/dank/models/hauhaucs-qwen36-35b-a3b-aggressive-fp8`
|
||
(35 GB).
|
||
|
||
## 2026-09-01 — dankserver media stack standardized to quadlet; dankclaw data roots migrated
|
||
- **Incident context**: Dankserver hard-crashed and rebooted 2026-09-01; every
|
||
bare compose-era media container (RomM trio, Jellyfin, Calibre-Web,
|
||
Copyparty) stayed dead at boot — the failure mode the Boot Persistence rule
|
||
exists for. Recovery doubled as the Phase 1 cutover.
|
||
- **Phase 1 (dankserver)**: six quadlet units shipped
|
||
(`dankserver/{romm,jellyfin,calibre-web,copyparty}/quadlet/`) + first-cutover
|
||
deploy handlers (retire bare containers → rescue RomM's anonymous `/romm`
|
||
volume → merge-move `/opt/<svc>` into `/home/dank/Containers/<svc>` →
|
||
ordered start with health gates). All six verified: units `active`,
|
||
`is-enabled=generated` (wants in the generator dir), ports 8085→8080,
|
||
8096, 8083, 3923 answering, jellyfin holds `/dev/dri` render nodes,
|
||
entropy binds provably ro except copyparty's `/w`. MariaDB continuity
|
||
proven via the real `dank` admin row (created 2026-08-31), not a fresh
|
||
bootstrap. Stale compose-era `container-*.service` generate-systemd units
|
||
retired to `~/.AGENTTRASH/20260901-stale-generate-systemd`.
|
||
- **Deployer findings fixed** (each caught live, each with a new seam):
|
||
v1 generator rejects `Command=` (→ `Exec=`), `Device=` (→ `AddDevice=`,
|
||
whole-unit rejection is silent), and splits spaces in `HealthCmd=`
|
||
(→ omitted for copyparty); `migrate_data_root` must merge into pre-seeded
|
||
destinations and rename mapped-subuid roots (100998) with `sudo mv`
|
||
(numeric-preserving, `keep` mode skips the chown — chowning mapped data
|
||
breaks the in-container users); quadlet unit dir must be created on a host
|
||
that never had quadlet; missing `PublishPort` = app answers inside the
|
||
netns only (copyparty 3923); seeded env files must carry **compose-provided
|
||
vars only** — the image-baked `HOME=/root` in a `[Service]
|
||
EnvironmentFile=` kills the podman client itself.
|
||
- **Phase 2 (dankclaw)**: data roots of gitea, authentik, hermes trios moved
|
||
to `/home/dank/Containers/<svc>/` while running (rename is inode-bound —
|
||
zero downtime), units repointed, redeployed through `deploy.sh --service`;
|
||
auth chain verified end-to-end (outpost ping → 302 to flow).
|
||
`deploy_hermes` ordering bug fixed: generated units must be reloaded
|
||
**after** unit files are copied, not before (stale ExecStart crash-looped
|
||
the trio on the old path).
|
||
- **Docs**: [[RomM]], [[Jellyfin]], [[Calibre-Web]], [[Copyparty]] updated to
|
||
the quadlet reality; [[Storage Policy]] now documents the Containers layout
|
||
+ the two ownership regimes; [[Deployment Flow]] service-manager table
|
||
corrected (dankserver media is no longer "bare containers, inventory only").
|
||
|
||
## 2026-08-30 (night II) — frank-npu deployed; NPU SVA/IOMMU root cause
|
||
- **Deployed** `frank-npu` on Dankclaw: FLM v1.0.1 (`/usr/bin/flm`,
|
||
dpkg `fastflowlm`) serving the whole local registry (36 tags) on port 52626;
|
||
primary model Gemma4 E4B IT NPU2 (vision + audio towers). First deploy
|
||
crash-looped every 10 s: `Open /dev/accel/accel0 failed (err=-19)`.
|
||
- **Root cause** (live-verified): the XDNA2 driver's SVA bind fails with ret -19
|
||
whenever the AMD IOMMU is off; the kernel cmdline carried `iommu=off` (a
|
||
perf/DMA tuning experiment from 08-20, applied at the 08-26 reboot). No IOMMU
|
||
groups ⇒ no PASID state ⇒ accel0 open returns ENODEV for every client. The
|
||
same root cause explains the `amdgpu ... no vmid pasid mapping supported`
|
||
kernel line — a consequence, not a GPU fault.
|
||
- **Fix**: `iommu=pt` in `GRUB_CMDLINE_LINUX_DEFAULT` + reboot (passthrough keeps
|
||
the iGPU default domain identity-mapped — zero DMA-translation overhead for
|
||
frank-large — while enabling IOMMU hardware + PASID, which SVA needs;
|
||
PASID domains are paged regardless of the default-domain type). Post-boot:
|
||
30 IOMMU groups, zero SVA-bind errors since boot, unit `enabled`,
|
||
`NRestarts=0` since boot.
|
||
- **Live-verified behavior**: whole-registry serving with per-request routing by
|
||
exact FLM tag (unknown tags silently serve `llama3.2:1b`); vision tower proved
|
||
end-to-end (generated red PNG as `image_url` data-URI → correct color answer,
|
||
279 prompt tokens consumed); audio tower proven with a control pair (440 Hz
|
||
tone via `input_audio` = 37 prompt tokens vs 15 for the identical text-only
|
||
control). Health probe is `/v1/version` — FLM has no `/health` route.
|
||
- `/scripts/frank-npu.env`, `/scripts/serve-frank-npu.sh`,
|
||
`/scripts/frank-npu.service` + deployer allowlist entries
|
||
(`c3ed3a6`, `cf0c541`) — service files and deploy wiring; unit is boot-persistent
|
||
(`WantedBy=multi-user.target`, enabled, `Restart=on-failure`, `RestartSec=10`,
|
||
`LimitMEMLOCK=infinity` — FLM requires unlimited memlock).
|
||
- **Docs**: new runbook [[frank-npu]] (linked from both indexes),
|
||
AGENTS.md fleet table + runtime section updated with the IOMMU/SVA hard
|
||
requirement. Client wiring: pi `models.json` registers the endpoint under the
|
||
Dankclaw provider with the exact FLM registry tag as model id.
|
||
|
||
## 2026-08-30 (night) — frank fleet double swap: STRIX on its pinned fork, Huihui on FreeToken
|
||
- **Dankclaw / frank-large**: swapped to
|
||
`kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF` on the card's
|
||
pinned `kingjones30/ROCmFPX` runtime (built at
|
||
`/home/dank/GitPackages/kingjones-ROCmFPX/build`, `LLAMA_BUILD_WEBUI=OFF`
|
||
— npm/HF-bucket asset provisioning aborts the link at 98% otherwise).
|
||
On-host 113.47 GiB file verified byte-identical to the HF LFS oid
|
||
(no re-download). Kept our args (401408×2, ub 512, fa on, fit off);
|
||
fa-on works UNPATCHED on this model — the QSA F32 assert was
|
||
agentionai-specific. No spec decode (card ships none; MTP heads
|
||
unloadable). Validated live: ~225 s load, health 200, reasoning
|
||
completions, real 111k-token prefill at 213–232 t/s. Deployed via
|
||
git flow; manual-process race during validation caught and fixed
|
||
(unit was crash-looping on a bound port while its health check
|
||
passed — check MainPID, not just /health).
|
||
- **Dankserver / frank-small**: swapped the model within
|
||
FreeToken v0.1.2 to `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8`
|
||
(native block-fp8, same loader path and pool plan; 35 GB downloaded
|
||
via `hf`, official checkpoint retained for rollback). Learned:
|
||
FreeToken's chat API 422s without an explicit `model` field. ~40
|
||
tok/s verified post-deploy. Both `.service` Descriptions refreshed.
|
||
- Docs: [[kingjones-strix-rocmfp4]] reactivated as the large runbook
|
||
(runtime build + swap procedure + validation), [[agentionai-fast-imatrix]]
|
||
marked RETIRED (rollback record), [[freetoken-frank-small]]
|
||
model sections updated, LLM index + AGENTS fleet tables refreshed.
|
||
|
||
## 2026-08-30 (late) — download chain proven end to end; qB credentials persisted
|
||
- **Cracked the arr↔Prowlarr feed contract**: route is `/{indexerId}/api`
|
||
(Prowlarr `NewznabController`; absolute `/api/v1/indexer/{id}/newznab`),
|
||
auth = the Prowlarr **global** key. The previously hand-rolled
|
||
`/torznab/{id}/feed` paths were Forms-auth catch-alls returning
|
||
`302 → /login` — which the arrs XML-parsed as "unexpected token '/'".
|
||
A source-verified librarian answer settled it in one shot.
|
||
- **Re-pointed all four arr indexer rows** (Radarr→YTS id7,
|
||
Sonarr/Lidarr→Nyaa id4, Readarr→Byrutor id3) via each app's own API;
|
||
every PUT passed the app's live connection validator (202); RSS enabled.
|
||
- **TV + music chains proven** by live search: Sonarr returned 76 current
|
||
releases via Prowlarr-Nyaa; Lidarr initially logged `0 active indexers`
|
||
(new rows default all-off) — enabling the search flags yielded 18
|
||
releases for YOASOBI – THE BOOK.
|
||
- **First real grab**: `MoviesSearch` → 3 YTS releases → *The Dark Knight
|
||
(2008) 2160p* (7.5 GiB) → qBittorrent 100 % → **imported**
|
||
(`hasFile: true`, 8,078,103,833 B at `/movies/The Dark Knight (2008)`,
|
||
hardlinked from the shared `/mnt/entropy` filesystem), queue empty.
|
||
- **Prowlarr Application auto-sync found broken by version skew**
|
||
(`Unable to cast object of type 'NzbDrone.Core.Thing…'` against Radarr
|
||
6.3) — the manual rows are the integration; documented.
|
||
- **qBittorrent permanent credentials installed**: the WebUI temp password
|
||
rotated on every boot (it had silently broken the arr clients once this
|
||
session). qb5 removed the API password endpoint and the lscr image has
|
||
no env support — wrote `WebUI\Password_PBKDF2` (source-verified
|
||
`base64(salt):base64(PBKDF2-HMAC-SHA512/100k)` form) into the conf with
|
||
the container stopped; login survives `podman restart` (204). All arr
|
||
download clients updated to it; value kept host-local in
|
||
`/home/dank/.qbpass` (0600), never in git.
|
||
- **Readarr parked with reason**: abandoned upstream; its qB client cannot
|
||
authenticate against qB 5 while three sibling apps pass with identical
|
||
credentials. Indexer row healthy; books pipeline awaits a successor.
|
||
- **Docs**: rewrote `Dankserver/prowlarr.md` (live wiring + six verified
|
||
gotchas incl. the joiner bug and the 302-masquerading-as-auth failure)
|
||
and `Dankserver/qbittorrent.md` (persistent-credential recipe, Referer
|
||
CSRF note).
|
||
|
||
## 2026-08-29 (late) — guarded-host sign-in fixed end to end
|
||
- **Diagnosed** both user symptoms: `Invalid next URL` = the flow being entered with an absolute cross-host `next=` (open-redirect guard rejects at completion); phone "hostname could not be found" = the public DNS zone had **no `auth` A record** (added via Porkbun API, `auth A 71.227.227.172 TTL 300`).
|
||
- **Disproved meta-header routing** live (full meta-header set still 404s; matching `Host` alone answers). The 2026.8 embedded outpost routes solely on the original `Host` vs each proxy provider's `external_host`.
|
||
- **Provisioned** proxy providers + applications for `search`, `hermes`, `prometheus`, `romm`, `files` (pks 2–6) via ORM, attached to the embedded outpost; `internal_host = http://caddy:8017` everywhere.
|
||
- **Wedge found + fixed live:** providers created with null `authorization_flow`/`invalidation_flow` starve the outpost's config fetch (`Failed to fetch outpost from API, retrying in 3 seconds` forever → entire outpost URL space 404s). Backfilled all three flows; the 3-second retry loop recovered without restart.
|
||
- `/dankclaw/Caddyfile` (`29f9a86`) — `:8016` is now a uniform whole-request `reverse_proxy authentik-server:9000` preserving `Host`; deleted the per-vhost redirect/flow design and stale comments.
|
||
- `/dankclaw/anubis/policy.json` (`2a72efa`) — ALLOW `^/outpost\.goauthentik\.io/` so login/transport paths are reachable before the challenge dance; app paths remain challenged.
|
||
- **Gotcha:** the policy bind-mount pins the file inode; a git checkout of a new policy file does NOT reach the running distroless container (observed 22 h of stale policy with zero reload logs). `--service anubis` deploy restart is mandatory after policy edits.
|
||
- `/tests/authentik.bats` — meta-header test replaced by the upheld contract (Host preserved; no absolute cross-host `next=` anywhere in the Caddyfile). Suite green (`0` not-ok).
|
||
- `/docs/Infrastructure/Authentik.md`, `/docs/Infrastructure/Anubis.md`, `/docs/Architecture/Network Architecture.md` (`ee1e038`) — rewritten to the live-verified mechanism: Host→provider routing, lockstep rule (guard set = TLS edge = :8017 table = provider list = public DNS), provisioning pitfalls, failure-mode rows for both symptoms.
|
||
- **Verified** by headless JS client: `https://search.franksdank.cloud/` → Anubis PoW solved → outpost intercept → authentik login form renders with the return path inside the signed OAuth2 `state` JWT. Credential submission is the user's step.
|
||
|
||
## 2026-09-14
|
||
- **Creation**: [[Hermes-WebUI]] — new `hermes-webui` compose service on Dankclaw (the Hermex server): runs `hermes-webui`'s `bootstrap.py --foreground` in the agent image, shares `/home/dank/hermes-data` with the gateway (identical rootless Podman uid mapping), publishes `127.0.0.1:8787` only, password-gated via `HERMES_WEBUI_PASSWORD` in the host `.env`.
|
||
- `/dankclaw/compose.yaml`, `/dankclaw/.env.example` — service definition and password placeholder; container-internal bind is `0.0.0.0` (Podman's port forwarder dials the bridge IP) while the host-side publish stays loopback so the SSH tunnel is the sole ingress.
|
||
- `/scripts/deploy-plan.py`, `/scripts/deploy.sh`, `/scripts/deploy-remote.sh` — `hermes-webui` registered as an isolated Dankclaw service in all three inventories; planner narrows edits to its compose block only.
|
||
- `/tests/hermes_webui.bats` — new seam: foreground entrypoint, loopback publish + auth, shared volume, `/health` healthcheck, env placeholder, planner narrowing (hermetic fixture repo), deployer dispatch.
|
||
- Remote access pattern documented: iPhone Termius local forward `127.0.0.1:8787 -> 127.0.0.1:8787` over the existing router port forward to Dankclaw `:22`; Hermex points at `http://localhost:8787`. No Tailscale, no Cloudflare.
|
||
|
||
## 2026-08-26
|
||
- `/dankclaw/Caddyfile` — Both Plex vhosts now proxy to PMS over a TLS upstream (`insecure_skip_verify`); PMS drops plain-HTTP connections from non-loopback sources, which produced empty-body 502s.
|
||
- `/docs/Infrastructure/Caddy.md` — Documented the Plex TLS-upstream requirement and the `/web/` vs `/` distinction.
|
||
- `/tests/caddyfile.bats` — Added a test pinning the TLS upstream transport on both Plex vhosts.
|
||
- `/docs/Dankclaw/Gitea.md` — Documented migration of Gitea web proxy ownership to canonical Caddy and retirement of the standalone proxy.
|
||
- `/docs/Dankclaw/Dankclaw Stack.md` — Updated boot-persistence and Caddy deployment descriptions for the two-unit Gitea stack.
|
||
- `/docs/Architecture/Deployment Flow.md` — Replaced the split-proxy safety description with the canonical Caddy ownership model.
|
||
- `/dankclaw/gitea/quadlet/gitea-caddy.container` — Retired the standalone Gitea Caddy Quadlet; deployment stages any live copy into home trash.
|
||
- `/scripts/deploy-plan.py` — Routes retirement of the legacy proxy to the canonical Caddy and removes it from the service inventory.
|
||
- `/scripts/deploy.sh` and `/scripts/deploy-remote.sh` — Removed standalone proxy dispatch and added idempotent retirement before canonical Caddy deployment.
|
||
- `/tests/caddyfile.bats`, `/tests/deployment_scoping.bats`, `/tests/deployment_flow.bats`, `/tests/frank_large_config.bats` — Added migration coverage and corrected the existing ubatch harness assertion.
|
||
- `/docs/index.md` — Bundle index remains the root navigation entry for the updated pages.
|
||
- **Boot-persistence audit (post-reboot, 2026-08-26 evening):** after Frank's reboot, Caddy and Pi-hole were both down. Caddy ran as a bare compose container (`restart: unless-stopped` does not survive shutdown's explicit stop; rootless Podman has no boot-time auto-start for bare containers). Pi-hole's quadlet existed but crash-looped (restart counter 1645): `net.ipv4.ip_unprivileged_port_start` was reset to 80 by two sources — a later-sorted sysctl drop-in (`99-rootless-ports.conf`) and a stale line in `/etc/sysctl.conf`, which systemd-sysctl applies **after** all of `sysctl.d/` (the first deploy pass only fixed the drop-in; the runtime value stayed 80 until the `/etc/sysctl.conf` line was neutralized too).
|
||
- `/AGENTS.md` — new MANDATORY section "Boot Persistence & Self-Healing": every deployed, proven-working container must have a self-healing, boot-persistent quadlet unit (`WantedBy=default.target` + enabled + linger, `Restart=always`, repo-provenance `.container`), verified before a service is declared deployed.
|
||
- `/dankclaw/caddy/quadlet/caddy.container`, `/dankclaw/caddy/env.example` — Caddy is now a boot-persistent quadlet (`caddy.service`): joins `dankclaw_default` so container-name upstreams keep resolving, `Pull=never` (local-only image), Porkbun credentials via `/home/dank/Containers/caddy/env` (deploy writes it from `.env`), bare-IP-catch-all health check. The caddy service was removed from `/dankclaw/compose.yaml`.
|
||
- `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` installs it to `/etc/sysctl.d/`, stages conflicting drop-ins (value > 53) in the home trash, comments out a stale `/etc/sysctl.conf` line (applied after all of sysctl.d), and applies `sysctl --system` (the boot code path).
|
||
- `/scripts/deploy-remote.sh` — `deploy_caddy` installs/starts the quadlet (retiring the bare container; all state is in host volumes). Boot persistence comes from podman's quadlet generator, which creates the `default.target.wants` symlink from the unit's `[Install]` section at every user-manager start — `is-enabled` reports `generated` for all healthy quadlets, and an explicit `systemctl enable` fails on generated units (learned live during the deploy). `deploy_pihole` enforces the sysctl prerequisites.
|
||
- `/tests/caddy_quadlet.bats` (new seam), `/tests/caddyfile.bats`, `/tests/deployment_flow.bats`, `/tests/pihole.bats` — quadlet boot-persistence, self-healing, network, env, and sysctl coverage; compose assertions repointed from the removed caddy service to the quadlet.
|
||
- `/docs/Infrastructure/Caddy.md`, `/docs/Infrastructure/Pi-hole.md`, `/docs/Dankclaw/Dankclaw Stack.md` — quadlet runtime facts, the 2026-08-26 incident, and the sysctl-conflict failure mode.
|
||
|
||
## 2026-09-01
|
||
- dankserver SSH recovered after the reboot (the "Not allowed at this time" port-22 responder vanished with it); the media stack had been restarted bare ~35 min post-boot, then the quadlet cutover completed for all six services (romm trio 13:45, jellyfin 13:49, calibre-web 13:50, copyparty 13:53 PDT) — units generated from the repo, data roots migrated to `/home/dank/Containers/<svc>`, `/opt/{romm,jellyfin,calibre-web,copyparty}` retired, all four endpoints answering, legacy `container-*` units gone. Dankserver deploy-state for the six services records deployed revisions through `1a257c0`.
|
||
- `/scripts/deploy-remote.sh` — frankbook handler fix: the credential-gate SKIP state now propagates Mac-side before restart+verify (a parked qbittorrent previously failed the batch and aborted the five arrs); qB first-boot seed writes a PBKDF2 `@ByteArray` WebUI password (plaintext is ignored by qB ≥ 4.2), sets `HostHeaderValidation=false` + `ServerDomains=*` for the socat forwarder path, and pre-creates the four categories with `save_path=/Volumes/COMPLETE/Finished Torrents`.
|
||
- `frankbook-pro` — full 7-service deploy run: gluetun + qbittorrent SKIP cleanly (WireGuard credentials pending — the Mac needs its own peer; reusing Dankclaw's key would flip its tunnel), prowlarr/sonarr/radarr/lidarr/readarr deployed, healthy, and reachable from dankserver over the LAN via the socat LaunchAgents (192.168.0.112).
|
||
- jellyfin GPU posture on dankserver: `AddDevice=/dev/dri` + `GroupAdd=keep-groups` in the unit (parity with pre-migration reality — no `/dev/nvidia*` in the container, no working VA-API driver in the image; the old bare container had the same. CPU/direct-play serves the household).
|
||
- `/docs/Infrastructure/FrankBook-Pro.md` (new), `/docs/index.md` — documented the Mac stack: VM placement, WireGuard gate, deploy flow, socat LaunchAgents, and the qBittorrent PBKDF2 seed.
|
||
- `/dankclaw/seerr/quadlet/seerr.container` — New Seerr (seerr-team/seerr v3.4.1) request-manager unit on Dankclaw: loopback-only publish 127.0.0.1:5055, SQLite under /home/dank/Containers/seerr/data, on dankclaw_default for the post-auth route.
|
||
- `/dankclaw/Caddyfile`, `/dankclaw/pihole/addn-hosts` — seerr.franksdank.cloud vhost (Anubis guard + :8017 @seerr post-auth handler) and LAN DNS record, deployed in lockstep with the Authentik outpost provider `seerr-proxy` (external_host https://seerr.franksdank.cloud/, internal_host http://caddy:8017) and the Porkbun A record 192.168.0.143.
|
||
- `/scripts/deploy-remote.sh` — deploy_seerr handler (install_quadlet + pre-create bind dir + restart).
|
||
- `/tests/seerr_quadlet.bats` — new seam: boot persistence, self-healing, pinned tag, loopback-only publish, data-root contract, appnet join, no env/envfile secrets.
|
||
- `/serverAppConfigs-docs/Dankclaw/Seerr.md`, `/serverAppConfigs-docs/index.md` — service doc + index entry.
|
||
- Lessons: rootless podman uid mapping is continuous (container N → host 100000+N-1; uid 1000 → 100999 — an off-by-one chown EACCES-crash-looped first boot); new quadlet bind-source dirs must pre-exist on target (statfs 125 otherwise); new vhost TLS issues at Caddy reload (~15 s dns-01), not at first request.
|
||
- Authentik outpost claim cache fixed (root cause of the persistent `/admin` 403 despite homepage-admins membership): the outpost serves per-browser-session claims from `authentik_providers_proxy_proxysession` until row expiry — akadmin's live row predated the group add and carried `["authentik Admins"]` only, and the 403 never generates an authentik event. Deleted his 4 rows; the next visit re-authorizes silently against the untouched authentik login session. Silent re-auth proven live: deleted qa-e2e-admin's 13 rows, ran the e2e admin spec against the stale storage state — 2 passed in 2.9 s, no login prompt. Mechanism + diagnosis SQL documented in `/serverAppConfigs-docs/Infrastructure/Authentik.md` (new "Outpost claim cache" section + failure-mode row); the doc's data-path references updated to the 2026-09-01 Containers layout.
|
||
- WireGuard credentials provisioned (Panama PA-12 conf → gluetun env inside the VM): tunnel egress verified (exit IP Panama, not residential), qB WebUI published per plan, and Prowlarr → qBittorrent wired — all five download clients (prowlarr, sonarr, radarr, lidarr, readarr) test 200 against live qB through the podman gateway `10.88.0.1:8080`; Prowlarr's client created via its `/downloadclient/schema` template (hand-minimal body 400s on create-time test).
|
||
- `/frankbook-pro/qbittorrent/quadlet/qbittorrent.container` — pinned `lscr.io/linuxserver/qbittorrent:4.6.7`: `latest` had resolved to qB 5.x, whose `204` empty-body login success Readarr's retired client rejects as "Authentication failure" at runtime (not just in API creates — the prior session's "runtime cookie auth unaffected" assumption was wrong); 4.6.7 answers the legacy `200` + `"Ok."`. Gotcha: after 5.x ran, its conf rewrite left a PBKDF2 line 4.6.7 rejects — every login `invalid credentials`; rewriting the same line fresh (same password, 100k iterations) fixed it.
|
||
- `/docs/Infrastructure/FrankBook-Pro.md` — wiring, pin rationale, PBKDF2 version-flip gotcha, and E2E result (test torrent through the Panama tunnel at ~17 MB/s, INCOMPLETE → COMPLETE into `Finished Torrents`, deleted with data).
|
||
- `/scripts/deploy.sh` + `/scripts/deploy-plan.py` + `/tests/dankserver_quadlet.bats` — committed earlier today (`b206f47`): dankserver quadlet standardization units/handlers/routing.
|
||
|
||
## 2026-09-04 — Sonarr moved to the admin side; internal-only by construction
|
||
- Operator directive (fixing an earlier seerr/sonarr mix-up): Sonarr is an
|
||
**admin-side** service and must be locally reachable only. The homepage
|
||
registry flipped its card from the user category to **admin** and repointed
|
||
the link at `http://192.168.0.112:8989` (LAN-direct). Users keep requesting
|
||
TV through Seerr, which stays on the user side behind the guard chain.
|
||
- Exposure closure: the unguarded public vhost `sonarr.franksdank.cloud`
|
||
(direct `reverse_proxy 192.168.0.112:8989`, no auth) and its post-auth
|
||
`@sonarr` handler were retired from the Caddyfile; the Pi-hole addn-hosts A
|
||
record for the name was dropped with it. No public Porkbun record ever
|
||
existed (NXDOMAIN at the authoritative zone), so after this change there is
|
||
no WAN path to Sonarr at all — manual DNS could previously reach the vhost,
|
||
which served the zero-auth UI to anyone who resolved the name manually at
|
||
the public edge; the vhost (added in `e0bade6`, 2026-09-04) is retired.
|
||
|
||
## 2026-09-15/16 — frank-large: ik_llama.cpp cutover + the decode campaign
|
||
- Deployed (0c1e10a, via clean-worktree stash cycle around the peer's
|
||
uncommitted files): halogen quadlet retired to
|
||
`~/.AGENTTRASH/20260915-ik-swap/halogen-quadlet`, frank-large now a root
|
||
systemd unit (`scripts/frank-large.service` → `/etc/systemd/system/`)
|
||
running the committed serve script. Service verified live: health 200,
|
||
reasoning completions, vision, slot persistence.
|
||
- Overnight performance campaign against the halogen bar (Frank: ~30 t/s
|
||
decode). Result: decode stuck at 3.8-4.3 t/s across every safe
|
||
configuration; prefill ~192 t/s = the CPU AVX512-VNNI ceiling. Full
|
||
evidence chain in [[frank-large-ik-uncensored]]: (1) ~100 CPU↔GPU backend
|
||
splits per token with the 3090 at 5% SM utilization — the decode wall is
|
||
GPU-section launch/sync overhead, not compute or memory placement;
|
||
(2) T1 ≈ T16 decode (3.97 vs 3.8-4.0) — the wall is a serial critical
|
||
path; (3) ik's CUDA-graph capture aborts on qwen4exp's per-layer
|
||
`ggml_repeat_4d` hc-broadcast ops; (4) mainline llama.cpp `37b53fd` runs
|
||
1.0 t/s on this model (CPU MoE kernels 4x slower than ik's iqk) — ruled
|
||
out; (5) the GGUF has no MTP head (mradermacher stripped it) — no
|
||
speculative decoding; (6) 84 GiB of experts cannot fit 24 GiB VRAM, so
|
||
halogen's all-resident 30 t/s is physically out of reach for this model
|
||
on this host. Re-evaluate when mainline's hc-ops + graph handling mature
|
||
(ggml-org PR #28901).
|
||
/- Measurement integrity traps hit live: the systemd unit's Restart=always
|
||
respawned pkill'd servers mid-experiment (phantom instances answering
|
||
health — manual experiments must `sudo systemctl stop frank-large.service`
|
||
first; plain `systemctl --user` lines are stale halogen-era docs); a
|
||
concurrent `-j16` build starved a decode measurement into a false timeout;
|
||
warm-cache `cat` benchmarks invalidated a concurrent perf window.
|
||
- `-ncmoe 42` (+6 GPU expert layers) measured 4.54 t/s decode at 22.0/24.6
|
||
GiB VRAM — documented as a tuning lever, not shipped (OOM margin at 262k
|
||
ctx is operator's call). `-ncmoe 48 -thp` at 4.6 t/s holds a 110 GiB
|
||
anonymous host buffer — co-tenant killer, never ship.
|
||
- Trash staged for approval: dankclaw `~/.AGENTTRASH/20260916-ik-partfiles/`
|
||
(111 GiB, the 3 GGUF download parts, superseded by the concatenated
|
||
verified GGUF), `~/.AGENTTRASH/20260915-ik-swap/` (halogen quadlet),
|
||
repo `tests/.AGENTTRASH/frank_large_quadlet.bats` (retired seam),
|
||
`scripts/.AGENTTRASH/frank-small*` (earlier retirement).
|
||
- Docs: `serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md` (new
|
||
runbook), LLM + root indexes relinked, AGENTS.md topology row +
|
||
Active Services section + bench-policy unit type. Committed 795b0b3;
|
||
tests 300 ok with the single pre-existing not-ok-73 (comfyui selector
|
||
bug, unrelated).
|
||
|
||
## 2026-09-16 — Dankclaw re-split: halogen returns as frank-large, llama.cpp takes the 3090 as frank-small
|
||
- `/dankclaw/frank-large/quadlet/frank-large.container` — halogen-flash-server
|
||
restored as the active runtime (the 09-15 ik swap never removed the file) and
|
||
retuned to the shape Frank asked for: `HALOGEN_CTX` 262144 → **200192**,
|
||
`HALOGEN_KV_POOL_POSITIONS` 401408 → **400384** (exactly 2 × 200192),
|
||
`HALOGEN_KV_SLOTS` 3 → **2**, `HALOGEN_MAX_TOK` 32768 → **24576** and
|
||
`HALOGEN_HOST_RESERVE_GIB` 15 → **20** (arena −4.2 GiB, for the co-tenant),
|
||
`HALOGEN_VISION_TOWER` **unset** (the tower moved to frank-small). Live ledger:
|
||
budget `121 − 67.7 − 20 = 33.5 GiB` against a need of `27.2`, so **no clamp**;
|
||
68.0 GiB weights + 11.0 KV + 16.5 working = 95.4 GiB held, 14.7 GiB left for
|
||
everything else (475 contiguous 2 MiB blocks). The co-tenant stall NOTE is
|
||
back, and the boot logged 4,350 compaction stalls (285 failed) — tight and
|
||
fragmented, not out of memory.
|
||
- `/scripts/frank-small.env`, `frank-small.service`, `serve-frank-small.sh` (new)
|
||
— mainline llama.cpp `37b53fd` on the RTX 3090 serving
|
||
`ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF` at IQ3_S: the `-mtp` build (the head is
|
||
`blk.64.nextn.*` in the SAME file, so `--spec-type draft-mtp` runs the draft
|
||
context on the main model's weights — no second copy, no `-md`) plus the BF16
|
||
projector, 131072 ctx / 1 slot, f16 KV, `--jinja` and `--metrics` pinned, port
|
||
7771, root systemd unit. 22,362 MiB of the card's 24,576 in use. Verified:
|
||
`/health`, chat with `reasoning_content`, a `tool_calls` round trip, slot
|
||
timing `draft acceptance 0.66–0.94`, and an image answered from the projector.
|
||
- `/scripts/deploy-remote.sh` — `deploy_llm_large` flipped back to halogen
|
||
(retire the ik ROOT unit into
|
||
`~/.AGENTTRASH/20260916-halogen-restore/host-units`, pull + re-tag the image,
|
||
**`wait_port_free 7770`**, install the Quadlet, start) and `deploy_llm_small`
|
||
added (root unit, model-file gate, health on 7771). The port gate exists
|
||
because the FIRST cutover attempt failed exactly there: pasta reported
|
||
`Address already in use` about a second after the ik server exited, the unit
|
||
died with code 126, and only `Restart=on-failure` recovered it. A redeploy now
|
||
stops this service's own quadlet before waiting, or the gate would deadlock
|
||
against itself.
|
||
- `/scripts/deploy-plan.py`, `/scripts/deploy.sh`, `/scripts/test-changed.sh` —
|
||
frank-small added to `SERVICE_ORDER` and the Dankclaw inventory (the stale
|
||
`dankclaw:frank-npu` entry dropped with it); the changed-test mapping for
|
||
`dankclaw/frank-large/*` pointed at `tests/frank_large_quadlet.bats`, retired
|
||
on 09-15, so the wrapper had been failing on that path ever since — now
|
||
`tests/frank_large_unit.bats`.
|
||
- Retired to `scripts/.AGENTTRASH/`: the ik trio (`frank-large.env`,
|
||
`frank-large.service`, `serve-frank-large.sh`), `chat-template.jinja` (only the
|
||
ik runtime read it), and four benchmark harnesses bound to retired runtimes.
|
||
- `/tests/frank_large_unit.bats` rewritten for the Quadlet (the `pool == 2 × ctx`
|
||
invariant, no vision tower, the port gate) and `/tests/frank_small_unit.bats`
|
||
added (root-unit shape, MTP-from-the-file contract, KV budget vs the card, port
|
||
separation). One assertion failed on its first run because the negative grep
|
||
matched a COMMENT in the serve script — the test now strips comments before
|
||
grepping the effective flags.
|
||
- `shared/homepage/services.json`, `shared/prometheus/prometheus.yml`,
|
||
`rules/llm-fleet.yml`, `grafana/.../llm-fleet.json` — a frank-small card at
|
||
`127.0.0.1:7771` and a `frank-small` scrape job at `192.168.0.143:7771`; the
|
||
`frank-large` job **removed**, because the halogen engine has no `/metrics` at
|
||
all (its routes are `/health`, `/cache` and `/v1/*`), so a target there would
|
||
have been red by construction; the dashboard's endpoint picker had been sourced
|
||
from a vLLM-only metric, so a llama.cpp job could never be selected. Live: 11
|
||
targets, 7 up, 4 down (comfyui, gitea, services — all structural), and all
|
||
three LLM homepage cards up.
|
||
- Harness (`~/.omp/agent/models.yml` + `config.yml`, dev box) — third provider
|
||
`llamacpp (dankclaw-3090)` for `:7771`; `frank-large` ctx 262144 → 200192 and
|
||
`input: [text]`; roles `smol`, `tiny`, `task`, `vision` → frank-small (the
|
||
image path and the small/fast roles), `slow`/`plan`/`designer`/`advisor` →
|
||
frank-large. Two failures found by driving it, both fixed: (1) frank-small's
|
||
template accepts only `xhigh|medium|low` for `reasoning_effort` and 500s on
|
||
anything else, and subagents run at `high` — every subagent on that endpoint
|
||
died until `compat.reasoningEffortMap` folded the levels onto that set; (2) the
|
||
ik runtime that had been serving `frank-large` rejected every request carrying
|
||
`tools` (`500 tools param requires --jinja flag`), which is what killed a
|
||
monitoring subagent earlier in the day. Both endpoints now answer tool calls,
|
||
and a subagent completes a tool-using turn.
|
||
- Docs: `frank-small-qwen38-gsq-rco.md` (new runbook),
|
||
`halogen-flash-server.md` re-activated with the 09-16 ledger,
|
||
`frank-large-ik-uncensored.md` marked retired, LLM index, deployment flow,
|
||
Observability and AGENTS.md topology/sections/lessons. Three AGENTS.md defects
|
||
found on the way: a truncated sentence mid-file ("There is **no audio"), a
|
||
harness path that does not exist (`/home/dank/.omp/agent/models.yml` — the
|
||
harness runs on the dev box), and the benchmarking stop/mask recipe still
|
||
describing frank-large as a root unit.
|
||
- Deploy: `./scripts/deploy.sh dankclaw --service frank-large --service
|
||
frank-small` — per-service health gates, then the e2e gate PASSED (31 passed,
|
||
1 flaky-retried). Suite: 323 seams ok, 0 failures.
|
||
|
||
## 2026-09-16 — The x4 pair: measured what a 3090 + Strix Halo combination can actually do
|
||
- Built mainline llama.cpp `37b53fd` with **both** the CUDA and Vulkan backends
|
||
(`build-dual`; glslc and libvulkan were already installed), to test the design
|
||
the earlier analysis only sketched: trunk + KV on the 3090, MoE experts on the
|
||
iGPU, PLE in LPDDR, one hop per boundary.
|
||
- Device map found: `CUDA0` = the 3090, `Vulkan0` = the same 3090, and
|
||
**`Vulkan1` = the iGPU** (RADV STRIX_HALO, `uma: 1`, 114 GiB GTT). The iGPU
|
||
being `Vulkan1` and not `Vulkan0` is exactly the detail that silently measures
|
||
the wrong device.
|
||
- Tensor inventory of the 111 GiB GGUF (mradermacher Flash-Next Uncensored):
|
||
experts **75.0 GiB**, `per_layer_token_embd` **32.8 GiB**, everything else
|
||
**3.2 GiB**; 12 of 48 layers are full attention (`[3,7,11,…,47]`), so only
|
||
they carry KV — at the engine's 28 KiB per position that is 5.5 GiB of KV read
|
||
per token at 200k context.
|
||
- Results (llama-bench, both services stopped per policy): iGPU-only (Vulkan,
|
||
PLE on CPU) **32.64 tok/s at ~0 depth, 25.77 tok/s at 64k depth, pp 456.7 →
|
||
231.8 into a 64k cache**, with the iGPU at 99–100 % busy; iGPU trunk + CPU
|
||
experts 12.10; 3090 trunk + CPU experts 20.46; and the cross-backend design
|
||
**aborted in the scheduler** — `pre-allocated tensor
|
||
(blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the
|
||
operation (NONE)` — while both per-layer cross-backend forms failed to load.
|
||
Mainline cannot serve one model from CUDA + Vulkan; the CPU is the only
|
||
fallback side that works.
|
||
- The comparison that matters: at fleet-realistic depth the iGPU-only run
|
||
matches halogen's field ~25 tok/s with the 3090 **idle** — but it commits 116
|
||
of the host's 121 GiB (78 GiB of Vulkan device buffers + 32.8 GiB of PLE page
|
||
cache) against halogen's 95.4 GiB held / 14.7 GiB spare, so it is not
|
||
deployable beside the fleet's ~28 co-tenants. Throughput parity, memory
|
||
bankruptcy.
|
||
- Also recorded: `GGML_CUDA_NO_PINNED` is mainline's pinning guard (ik called it
|
||
`GGML_CUDA_NO_PINNED_WEIGHTS`); `systemctl mask` is refused for a relinked root
|
||
unit (frank-small's unit is a symlink into the repo); and llama-bench's `tg`
|
||
instances run with `n_prompt = 0`, so `-d` is required to put KV into the
|
||
measurement at all. Both services were stopped and restored (both healthy;
|
||
halogen's ledger read `400384 positions, ~27.2 GiB` against a 33.5 GiB budget
|
||
with 15.2 GiB left).
|
||
- New page: [[x4-pair-experiment]].
|
||
|
||
## 2026-09-16 (later) — the x4 pair, round two: the first conclusion was a parsing artifact
|
||
- Asked whether **pure Vulkan** changes the answer (both devices can run Vulkan —
|
||
`Vulkan0 = RTX 3090 (NV_coopmat2)`, `Vulkan1 = RADV STRIX_HALO (KHR_coopmat)`).
|
||
It does not: the per-tensor design still aborts, byte-identically.
|
||
- **The real gate is `-ot`, not the driver mix.** `has_tensor_overrides()` is a
|
||
required term of `pipeline_parallel` in `llama-context.cpp:433` — the ONLY place
|
||
in the tree that reads it — and that flag is what sets `n_copies` in
|
||
`ggml_backend_sched_new` (`ggml-backend.cpp:25`). With `n_copies == 1` a
|
||
pre-allocated weight on a non-primary device has no backend, so the graph
|
||
aborts at `ggml-backend.cpp:941` with the leaf's own op name (`(NONE)`).
|
||
`-ot exps=CPU` works only because the CPU backend is last in the list.
|
||
- **Then the correction.** `llama-bench --help` says `-dev <dev0/dev1/...>`: `/`
|
||
splits a device LIST, `,` splits SWEEP values. So `-dev CUDA0,Vulkan1` was two
|
||
*single-device* runs — the earlier "CUDA + Vulkan cannot serve one model", the
|
||
"load failures", and the OOMs were artifacts (the OOM was 111 GiB on the 24 GiB
|
||
card). The bench table's `dev` column had been printing the truth all along.
|
||
- **With the correct syntax two-device splits work, in both flavours** (64k depth,
|
||
`-d 65536`; placement proven by `nvidia-smi` sampled during the run):
|
||
- pure Vulkan `-dev Vulkan1/Vulkan0 -ts 0.85/0.15` — 12.8 GiB on the 3090,
|
||
**27.55 tok/s** (pp512@64k 224.3)
|
||
- cross-vendor `-dev CUDA0/Vulkan1 -ts 0.15/0.85` — 15.0 GiB on the 3090,
|
||
27.23 tok/s with **pp512@64k 268.5**
|
||
- against the iGPU alone: 25.77 tok/s / 231.8 — so **+7 % decode, +16 % prefill**
|
||
- **And the split still cannot reach the design's premise**: whole layers move
|
||
their experts with them (1.62 GiB per layer), the 12 attention layers are spread
|
||
every 4th across the depth, and the 32.8 GiB PLE must stay on the main device —
|
||
so KV cannot be consolidated on the 3090.
|
||
- **Incident, same evening:** the rootless podman pause process died — trigger
|
||
NOT established (no OOM kill in `journalctl -k` for that window, so "the
|
||
116 GiB runs squeezed it out" is a guess, not a finding). Every `podman`
|
||
command then failed (`invalid internal status … could not find any running
|
||
process`), every container start failed on aardvark-dns binding `10.89.x.1:53`
|
||
(the rootless netns holding the bridges was gone), and all 28 quadlets looped.
|
||
`podman system migrate` — podman's own named remedy — **panics** in 5.7.0
|
||
(`teardownNetworkBackend`). Repair was a **reboot**: all 28 containers returned
|
||
on their own (linger + `WantedBy=default.target` present), e2e gate green after.
|
||
- **A second, pre-existing defect surfaced by that reboot:** `hermes.franksdank.cloud`
|
||
returned 502. hermes-webui is `Network=host`, so 8787 is a bare host-netns bind
|
||
and netavark DNATs only *published* ports — from inside the Caddy container
|
||
`192.168.0.143:8787` and `10.89.1.1:8787` are refused while
|
||
`host.containers.internal:8787` (169.254.1.2) answers 200. Fixed in the
|
||
Caddyfile, the stale claim in its comment replaced with the measurement, and the
|
||
seam that guarded it **strengthened** (it accepted any numeric upstream, so the
|
||
LAN IP had been passing it; reverting now fails it — teeth verified).
|
||
- Both endpoints restored and healthy: frank-large `400384 positions, ~27.2 GiB`
|
||
against a 33.5 GiB budget (no clamp), frank-small 200 on 7771. Page:
|
||
[[x4-pair-experiment]] (rewritten).
|
||
|
||
## 2026-09-16 (night) — model cache cleared on instruction, then re-fetched, and the update was real
|
||
- **Operator instruction:** purge the model cache outright so no run can serve a
|
||
stale payload. Done with one-time `rm` authorisation, and the two *live*
|
||
payloads were cleared and immediately re-fetched — because deleting them
|
||
without a re-fetch means neither endpoint can start.
|
||
- **Cleared:** the Uncensored i1-Q4_K_M payload (112G), the Q4_K_S part-fragments
|
||
(105G), the retired kingjones STRIX and agentionai ROCmFP4 payloads (114G +
|
||
863M), an unreferenced Qwen3.5 vision dir (8.8G), and all of `~/.AGENTTRASH`
|
||
including the ik part-files (119G). Disk went **298 GB → 895 GB free**; 814 GB
|
||
after re-fetching what the services need.
|
||
- **The re-fetch produced genuine updates.** halogen's sidecars had been stale —
|
||
`overlay.hgn` 2,477,677,120 → **2,572,466,560** B, `overlay-speed.hgn`
|
||
2,383,306,048 → **2,478,095,488** B — while `w4b.hgn` came back the same size
|
||
(124,068,083,904). So "clear and re-pull" *is* an update mechanism here, and
|
||
the sidecars are versioned **in place**: a stale copy is invisible from the
|
||
file name. frank-small's 27B + mmproj re-fetched to hashes matching the Hub
|
||
exactly and are unchanged.
|
||
- **Trap:** podman refuses a bind mount whose source directory is missing
|
||
(`Error: statfs /home/dank/models/halogen: no such file or directory`), so
|
||
frank-large looped on `Restart=on-failure` until `mkdir -p` recreated the
|
||
**empty** directory — then the engine fetched the repo itself
|
||
(`HALOGEN_DOWNLOAD`), which is the documented refresh path.
|
||
- **Q4_K_S vs Q4_K_M, measured on the same host and build:** +2.1 / +1.6 / +2.6 %
|
||
(iGPU-only pp512 / pp16384 / tg64), +2.7 % tg32@64k, +4.1 % and +2.0 % for the
|
||
two layer-split configs, ~0 on both pp512@64k. Host RAM bottomed at **10.0 GiB
|
||
available** in A\* against ~5 GiB for Q4_K_M. A memory lever with a small speed
|
||
dividend — nothing structural moves.
|
||
- **IQ4_XS pulled in parallel with the bench**, as asked: 97,473,155,360 bytes,
|
||
sha256 `b4787265…d0574f9d`, matching the Hub's LFS digest. Kept as the fresh
|
||
payload for the next comparison; Q4_K_S was cleared once the bench was done.
|
||
- **Held back deliberately:** `models/checkpoints` (23 GB of Flux 2 Klein — no
|
||
unit mounts it, but its upstream is licence-gated, so it is not mine to
|
||
destroy) plus the small tool dirs under `models/` and `models/logs`, which is
|
||
the evidence for this page.
|
||
- **Verified after the refresh:** `/v1/models` correct on 7770/7771, real
|
||
completions returning `pong` on both, halogen's ledger `68.0 GiB weights
|
||
locked / 11.0 GiB KV pool`, 2 slots over one 400384-position pool, and **307**
|
||
contiguous 2 MiB blocks left (240 before the refresh).
|
||
- Runbook: [[halogen-flash-server]] gains a "Refreshing the weights" note; the
|
||
quant comparison is appended to [[x4-pair-experiment]].
|
||
|
||
## 2026-09-17 — frank-small swapped to Gemma4-12B QAT (Q4_K_M)
|
||
|
||
- **Cutover:** `frank-small` (Dankclaw RTX 3090, :7771) replaced
|
||
ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with
|
||
`HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` Q4_K_M —
|
||
7.38 GiB main + 175 MiB BF16 mmproj (with audio encoder) + 254 MiB
|
||
`mtp-gemma-4-12B-it.gguf` assistant sidecar, all sha256-verified.
|
||
- **Spec decode:** the MTP head is now a separate `gemma4-assistant`
|
||
file loaded via `--model-draft` (shares the target's last layers' KV,
|
||
`llama-model.cpp:2662`), chained behind draftless `ngram-mod`:
|
||
`--spec-type draft-mtp,ngram-mod`. Measured acceptance 0.38–0.46,
|
||
mean draft len ~2.2.
|
||
- **KV:** q8_0 K and V; `build-cuda` rebuilt with
|
||
`-DGGML_CUDA_FA_ALL_QUANTS=ON` (required, else ~12x prefill collapse).
|
||
`CTX_SIZE` unset → `--fit on` chose the full 262,144-token window;
|
||
idle card usage 11,996 / 24,576 MiB (was 22,362).
|
||
- **Template behavior:** Gemma4 accepts every `reasoning_effort` level
|
||
(all seven probed: 200) — the Qwen3.8 gate is gone; harness map kept
|
||
as a safety net.
|
||
- **Verified live:** health 200, chat + `reasoning_content`, `tool_calls`,
|
||
vision (live JPEG), deploy e2e gate passed (31 passed, 1 flaky crawl).
|
||
- **Harness:** `models.yml` frank-small `contextWindow: 262144`.
|
||
- **Note:** the boot log's `Gemma4Assistant requires ctx_other to be
|
||
set` error is `--fit`'s own probe — normal, self-recovers seconds later.
|
||
- Runbook: [[frank-small-gemma4-12b]] (active);
|
||
[[frank-small-qwen38-gsq-rco]] retired 2026-09-17 as rollback target.
|
||
|
||
## 2026-09-17 (later) — frank-small: four full windows, max-length throughput sweep
|
||
|
||
- **Slot ceiling measured:** 1 full-ctx slot idles 11,996 MiB, 4 idle
|
||
22,346 MiB -> 3,450 MiB/slot, ~8,546 fixed; a 5th projects past the
|
||
24,576 MiB card. Deployed `CTX_SIZE=1048576` / `PARALLEL=4` (each
|
||
slot the FULL 262,144 window; explicit CTX bypasses `--fit` on
|
||
purpose — fit would divide the pool at parallel>1). The bats seam now
|
||
pins the division and the budget inequality.
|
||
- **Audio verified:** JFK WAV via `input_audio` -> verbatim transcript.
|
||
- **Batch sweep at max length** (salted API probes, pp ~254k /
|
||
tg @ ~253k ctx): ubatch 512 -> pp 912–924, tg 90–204;
|
||
**ubatch 1024 -> pp 962–965, tg 226–302 (deployed)**;
|
||
ubatch 2048 -> OOM in the MTP draft's pp compute buffer (~3.3 GiB
|
||
vs 2.2 GiB headroom).
|
||
- **Negative results kept:** draft n-max 4 + poll 100 decoded WORSE at
|
||
250k (230–257) — longer verification rounds re-read more long-ctx
|
||
attention than the accepted token earns. No CUDA-graph flags exist in
|
||
this build, so launch-overhead tuning is off the table.
|
||
- **Concurrency caveat:** a 254k prefill beside a decode drops it
|
||
~240 -> ~100–135 t/s; one engine, no per-slot QoS.
|
||
- **Housekeeping during the sweep:** the earlier live service was
|
||
discovered to be an orphaned manual boot holding 7771 past the unit
|
||
stop (kill -9'd); the sweep now refuses to measure a port it does not
|
||
own. Peer frank-medium WIP was stash-held around the deploy and
|
||
self-committed as `23f0978`/`74dda30` — nothing lost.
|
||
- **Live after deploy:** `n_slots = 4, n_ctx_slot = 262144`, idle
|
||
23,300 MiB, e2e gate 32 passed.
|