- runtime: charlie build-qwen38 -> kingjones30/ROCmFPX (card recipe
build, LLAMA_BUILD_WEBUI=OFF since npm/HF-bucket provisioning fails;
all 35 of our serving flags verified present in the tree)
- model: agentionai FAST imatrix -> kingjones777 STRIX single GGUF
(on-host copy sha256 == HF LFS oid c6770d74...49fa verified)
- args unchanged from our validated profile (401408x2, b4096/ub512,
t16/tb16, f16 KV, fa on, fit off); mmap stays default per card
(PLE streams via page cache; --no-mmap = silent cgroup OOM)
Meta-header routing and forward_auth removed from the described design — both
disproved live. Documents: Host-based provider routing with per-guarded-host
proxy providers (lockstep rule), the null-flows wedge that starves the
embedded outpost config fetch, the Anubis /outpost.goauthentik.io/* transport
exemption, the single-file bind-mount stale-policy inode gotcha, Invalid next
URL as a dead design, and the public-DNS record requirement per guarded host.
Login/PoW/POST traffic to the outpost URL space cannot solve a fresh PoW
challenge per request; exempt the transport prefix (authentik's own session
gate still protects it) so the guarded-host sign-in journey completes.
The redirect-to-flow design with an absolute cross-host next= was rejected by
the flow's open-redirect guard at completion ("Request has been denied:
Invalid next URL"), and meta-header routing does not exist in this build
(providers route on the ORIGINAL Host). :8016 now proxies every request whole
to authentik-server:9000 preserving Host; the embedded outpost serves login
in place and forwards authenticated traffic back to :8017. Requires one proxy
provider per guarded host (provisioned live: search/hermes/prometheus/romm/
files + www, flows backfilled). Tests flipped from the meta-header claim to
the upheld contract: Host preserved, no absolute next= redirects.
fa on aborted graph build: build_attn_qsa called ggml_fill on the
F16-converted kq mask (GGML_ASSERT F32 at ggml.c:5371). Patch mirrors the
MLA sparse path in llm_graph_context::build_attn (F32 base mask for the
top-k surgery, F16 cast only at the attention call, gated on
cparams.flash_attn). Vendored at scripts/patches/qwen4exp-qsa-f32-mask.patch
(sha256 f2f4a486...); the build-qwen38 binary on Dankclaw is rebuilt with
it and validated live at CTX_SIZE=32768 (health 200, real completion).
- Main model (v2 joined PLE layout, sha256 1e4160ca…) loads and serves on
build-qwen38 only with --flash-attn off: the QSA indexer path calls
ggml_fill on the F16-cast kq mask and aborts graph build (GGML_ASSERT
F32 at ggml.c:5371). Verified live: model loaded, health 200, real
completion.
- MTP draft head (agentionai FAST, 2.28 GiB) does not load on this binary:
qwen4exp loader requires full per-layer blk.<il>.hc_attn_norm.weight,
head carries only blk.48. Third confirmation (EasiiX Q8_0 + agentionai
FAST vs kingjones STRIX_LEAN previously, now own-matched pair). Spec
decoding runs ngram-mod only; head retained for a future compatible build.
- FLASH_ATTN=off and SPEC_TYPE=ngram-mod wired through env + serve script;
runbook and AGENTS.md updated with the verified constraints.
Model: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF (87.06 GiB,
Q4_0_ROCMFP4_FAST imatrix-calibrated, per-head PLE table inside the GGUF —
fully VRAM-resident). Draft: agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST
(2.28 GiB, matched ROCmFP4-FAST quant, ~66% acceptance at n-max 3).
Vision tower (f16) loaded via mmproj/.
Runtime: charlie12345/ROCmFPX build-qwen38 (HIP, ROCm 7.2.4, gfx1151) —
the only binary with the ROCmFP4 FAST types. Replaces the jcbtc CIRU v1.1
deployment (ciru-ai runtime).
Profile: card sampling table (temp 0.7, top_p 0.8, top_k 20, min_p 0.0,
presence 1.5) mixed with the validated build-qwen38 serving profile:
262144 ctx x 1 slot, b 4096 / ub 4096, t 16 / tb 16, f16 KV, fa on,
cont-batching, prompt cache + 8 GiB cache-ram + idle slots, 16 ctx
checkpoints (no --checkpoint-min-step — unsupported on this binary),
--fit off (auto-fit segfaults on qwen4exp hyper-connection config),
--spec-type draft-mtp,ngram-mod with n-max 3 (adaptive p-split measured
worse per author card), p-min 0.75. Qwen-Sharp chat template kept;
reasoning stays in DeepSeek format.
The 2026-08-28 failure of this draft head on the kingjones STRIX_LEAN
model was a cross-vendor pairing problem: the head carries
blk.48.hc_attn_norm.weight (verified in-GGUF, same shape as the working
CIRU draft). Docs: new runbook agentionai-fast-imatrix, index updates.
Runtime: ciru-ai/Qwen3.8-Flash-CIRU-STRIX-IU4 llama.cpp at tag v1.1, built
with the vendor build-linux-amd.sh flag set plus GGML_CUDA_FA +
GGML_CUDA_FA_ALL_QUANTS (the Q4_1_MOE_FORCE_J path is live in the hipified
mmq kernels). Model: jcbtc/Qwen3.8-Flash-CIRU-STRIX-IU4 (Q4_1 IU4 experts,
MTP Q8_0 draft, CIRUPLE1 PLE sidecar) with checksums verified at download.
Vendor production profile verbatim on first deployment: 262144 ctx x 1 slot,
b 2048 / ub 512, t 8 / tb 8, f16 KV, fa on, cont-batching, prompt cache +
8 GiB cache-ram + idle slots, 32 ctx checkpoints (min step 8192), MTP depth
3 with p-split 0.10, PLE pager env knobs (Q41_MOE_FORCE_J=32,
PLE_WORKERS=16, PLE_STRICT_SHA=0, HIPBLASLT=1). Qwen-Sharp chat template
kept deliberately; reasoning stays in DeepSeek format.
This 2026.x build does not mount /outpost.goauthentik.io/* for proxy-provider
meta-header routing (probed live: ping answers 204, app paths with full meta
headers 404). Guarded vhosts now redirect to /if/flow/default-authentication-flow/
with next= back to the app — unauthenticated sees the login page in place,
authenticated sessions complete instantly and bounce straight to the app.
The proxy provider matches external_host exactly, so Host=www matched zero
providers and Authentik answered its error page while the naked domain
served the homepage. Flip the topology: www.franksdank.cloud is the one
guarded vhost carrying the homepage through Anubis -> Authentik -> :8017;
franksdank.cloud (plus legacy names and the bare-IP catch-all) 308-redirects
onto www before any challenge. Tests flipped to assert the new contract.
All four arr roots live: /tv, /movies via API; /music, /books created after
v1 validator layers peeled back (quality/metadata profile ids must reference
existing profiles; v1 has no /api/v1/profile endpoint). Auth pitfall: the
literal AuthenticationRequired=Disabled string throws DryIoc.ScopedItemException
in v1 AuthorizationMiddleware on every request — the valid no-auth state is
ABSENCE of the key. Fixed live on four containers, verified boot clean.
Sonarr/Radarr root folders created via API (201). Lidarr/Readarr deferred to
first-run wizard (v1 validators demand quality-profile ids that don't exist
yet). Root cause of the write failures: LSIO PUID=1000 under rootless podman
maps to host uid 101000 — a phantom user owning nothing; PUID=0 maps to the
host user that owns every media dir.
Under rootless podman, LSIO's PUID=1000 maps to host uid 101000 — a phantom
user owning nothing: Sonarr's own validator rejected /tv because the process
could not write it. Container-root under a single-user userns maps to the host
user (dank), which owns every /mnt/entropy media dir and /opt config dir.
Identity is now mapping-independent instead of depending on subuid offsets.
calibre-web untouched (working; churn avoided).
Verified live: indexers->Prowlarr->per-arr QBittorrent download clients with
category routing. Readarr's client parked (qB proxy auth defect in abandoned
upstream). EZTV/1337x parked with reason (CloudFlare-blocked at this egress);
Nyaa/sukebei/RuTor/Byrutor/YTS passed live tests and are enabled. Prowlarr v2
Applications rows now carry real arr baseUrl+apiKeys; before this every sync
attempt threw NullConfig in GetIndexerMappings.
Verified live: AuthenticationMethod=None + AuthenticationRequired=Enabled on
all five arrs; unauthenticated API probes return 401 both locally and through
the vhosts. Also records the as-built divergences (entropy mounts, LA timezone,
readarr tag pin, prowlarr mount fix) that supersede the design compose.
Kestrel rejects the h2c upgrade preface; Caddy surfaced that as a 502 per
request while the backend answered 200 on plain HTTP/1.1 from Caddy's network.
Contrast recorded: Plex PMS needs the TLS transport, Jellyfin needs none.
Jellyfin's Kestrel backend rejects the h2c upgrade preface, so Caddy turned
every request into a 502 while the same port answers 200 over plain HTTP/1.1
from Caddy's network. Default transport (HTTP/1.1) restores the vhost.
The page claimed an env-var interface (COPAY_PASS/COPAY_MODE) and a /data
volume convention that this image does not implement: the ac edition consumes
accounts and volumes as argv flags (-a USER:PASS, -v SRC:DST:PERMS), serves
the LUKS array at /w, and answers HTTP 500 with its own error page when zero
volumes are configured. Image ref corrected to copyparty-ac (bare latest tag
does not exist on ghcr).
Ground truth from the image itself: python3 -m copyparty --help-accounts shows
-a takes USER:PASS and -v takes SRC:DST:PERMS:FLAGS — the COPAY_PASS/COPAY_MODE
envs this compose carried are not consumed by this image at all (zero COPAY_
references in the package). With no accounts and no volumes configured, the ac
edition serves its cwd read/write for everyone: the stack was running an
unauthenticated world-writable share of the 91T array. Now: admin account dank
with the A flag on volume /w; anonymous gets nothing.
The ac edition's entrypoint (cpp.sh -> python3 -m copyparty) consumes config
via /z/initcfg, not argv; -v is the volume flag and ate the next arg,
crash-looping argparse. The image mounts everything under /cfg/copyparty as
volumes by marker-file convention — plant the marker on the host instead.
The image entrypoint is /bin/ash /z/cpp.sh -c /z/initcfg: copyparty flags are
passed through command, and the ac edition mounts volumes with -m, not via a
COPAY_VOL env. Without any volume mounted every request resolved zero volumes
and the app answered HTTP 500 with its own error page. Verbose flag included
for the bring-up; drop it once stable.
The ac edition maps volumes from COPAY_* envs; without a volume root every
request resolves zero volumes and the app answers HTTP 500 with its own error
page. Point it at /w where the LUKS array is mounted.
ghcr.io/9001/copyparty:latest does not exist — ghcr answers 403 on the token
request for nonexistent repos, which is why the bring-up failed at pull. The
upstream publishes ghcr.io/9001/copyparty-ac (ac = full edition) and serves
whatever is mounted at /w, so the compose mounts the LUKS array there; config
persists in /opt/copyparty/cfg per upstream layout.
Jellyfin ran as user 1000:1000 inside a rootless user namespace, where host
uid 1000 maps to container uid 0 — the pinned uid is unmapped, so /config/log
creation EACCES'd and the service SIGSEGV'd on boot. Container root (= host
dank) is correct under rootless podman; group_add 992 (render) still maps via
the userns group map, so GPU access is preserved.
Readarr: lscr.io/linuxserver/readarr:latest does not exist — LSIO ships only
arch-prefixed nightly/version tags for readarr (verified against the ghcr tag
API). Pin amd64-1.0.980, the newest stable tag.
The compose stack mounted /mnt/media/{movies,shows,tv,music,books,downloads}
— a prefix that has never existed on this host. The 91T LUKS array mounts at
/mnt/entropy and its libraries are case-exact (Movies, TVShows, Music,
eBooks). Also restores prowlarr's config mount (eaten by an off-by-one in the
prior edit) and fixes TZ New_York->Los_Angeles on the arr stack.
Live-verified 2026-08-29: plexmediaserver.service is an enabled native
systemd unit on Dankserver (not a quadlet; updates via apt, not deploy.sh),
and tautulli runs as a snap with no auth wall on 0.0.0.0:8181. The hunt list
named both; neither had a page. Index linked per the docs-completeness seam.
The policy must ALLOW Anubis's own endpoints before the challenge-all rule
or the challenge challenges itself: browsers POST-solve via make-challenge
then GET-navigate pass-challenge, and a challenged answer renders as the
browser's own fetch-failure error page. Non-JS clients never reach the
verify step, which is why every curl probe looked green while every human
hit a wall. Also records that v1.13.0's valid action verbs are
ALLOW/CHALLENGE/DENY (ACCEPT crash-looped the unit on first deploy).
The pinned binary's policy parser rejects unknown actions and exits 1 on
startup (verified against the image binary: the valid action literals are
ALLOW, CHALLENGE, DENY). The first cut of the own-endpoints accept rule
used ACCEPT and crash-looped the unit on deploy.
The catch-all path_regex "." policy challenged everything, including
Anubis's own API namespace /.within.website/x/cmd/anubis/*
(make-challenge, pass-challenge, static assets). In whole-request proxy
mode a JS-capable browser POSTs its challenge solution to
pass-challenge and received another challenge page (HTML) instead of the
verification verdict (JSON): the challenge app's fetch failed and the
browser rendered its own error — 'the request filters did not return an
alternative response'. Non-JS clients never POST the solution, so every
curl probe looked green while every human hit a wall.
Accept rule ordered above the catch-all; policy is volume-mounted ro from
this checkout, so the deploy updates it in place.
The forward_auth target /outpost.goauthentik.io/auth/caddy does not exist
in authentik 2026.8 (verified live: 404 for every header combination; the
URL tree has no outpost auth prefix at all — the outpost worker dials back
to core over ws/outpost/<uuid>/ instead). Unauthenticated subrequests got
404, Caddy treated >=400 as 'not authenticated' and redirected browsers to
the dead path — users saw authentik's own Not-Found page.
Fix: guarded vhosts now proxy the whole request through authentik-server:9000
with X-authentik-meta-outpost/-provider/-app headers (the exact header set
this build's Traefik controller emits, adapted for Caddy). Unauthenticated
clients get the login flow served in place; once a session exists,
authentik proxies on to the provider's internal_host — now
http://caddy:8017, a new post-auth internal listener that routes straight to
the app. The chain terminates; nothing dials back into authentik.
Live DB changes applied alongside (postgres is the source of truth):
provider pk=1 authentication_flow=default-authentication-flow (was NULL —
the actual cause of the 404s) and internal_host=http://caddy:8017 (was
self-referential http://authentik-server:9000).
The quadlet-dir allowlist silently dropped every change under
dankclaw/authentik/ — the new outpost unit (and its siblings) now route
by exact unit name, mirroring the searxng/gitea pattern. Adds
authentik-outpost to the service inventories; it is a quadlet unit, not
a compose service, so it stays out of the compose set.
In authentik 2026.8.x the proxy-provider URL space (/outpost.goauthentik.io/*)
is served by the Rust worker binary, not the server — verified live: with a
tenant-matching Host header, /if/flow/... answers 200 on the server while
/auth/caddy still 404s. The outpost instance existed in the DB but no process
ever served it, so Caddy's forward_auth subrequests hit a 404 and browsers got
authentik's own Not-Found page instead of the authentication flow.
The new unit runs the same image as server/worker with Exec=worker and flips
into outpost mode via AUTHENTIK_HOST/AUTHENTIK_INSECURE/AUTHENTIK_TOKEN (the
token is read from the DB into the host .env at deploy time; it never enters
the repo). It dials back to core over websocket and serves forward-auth on its
own 9000. Deploy handler restarts only this unit — the rest of the stack is
untouched.
Replaces the pre-Anubis direct-routing diagram with the live chain:
Caddy TLS -> Anubis :8923 verify -> accepted traffic back to Caddy's
internal listener :8016, where per-vhost routing resumes and Authentik's
embedded outpost gates wrapped apps via forward_auth. Guarded set
(homepage, search, hermes, prometheus, romm, files) and the
exempt-by-construction set (auth, plex, gitea git paths, grafana) are
listed explicitly. Access Policy bullet updated: basic auth was replaced
by the challenge-then-forward_auth layering.
Three compounding deploy-handler bugs found by live deployment:
- podman image exists/pull used a mixed-case ghcr.io ref that podman's
reference parser rejects before any network I/O (repository name must be
lowercase); all refs now lowercase — ghcr resolves case-insensitively.
- the unit runs the image under a localhost/ alias tag: this host's podman
5.7 quadlet path fails container create with 'image not known' for
registry-prefixed refs even when present in storage (verified live).
- the health probe was shell-form and the distroless ko-build image has no
shell; HealthCmd is now a JSON exec-form array, which passes through the
quadlet generator's argv assembly verbatim.
Tests updated to assert the alias-tag reality plus the pull guard.
The image has no shell (ko-build distroless; entrypoint /ko-app/anubis),
and this host's quadlet generator joins a bare HealthCmd into one
\x20-escaped argv element that podman then runs shell-form — exec of a
missing shell exits 1 and the container sits unhealthy. A JSON array in
HealthCmd passes through the generator verbatim and executes the binary
directly.
The v1.13.0 image is distroless: no shell, no HEALTHCHECK in the image
config (verified live: Healthcheck config is null on this host's image),
so quadlet inherits nothing. Declare the probe explicitly in exec form;
'anubis --healthcheck' exits 0 when the challenge server answers.
On this host's podman 5.7 quadlet path, a registry-prefixed Image ref
fails to resolve from local storage at container create time even when
the image is present ('image not known'); the identical argv works from
a shell or transient unit. Verified live: an identical unit with
Image=localhost/techarohq/anubis:v1.13.0 starts immediately. The deploy
handler now tags the alias after the guarded pull; upgrades re-tag
after pulling the new tag.
image inspect on an absent reference can exit non-zero for reasons
unrelated to existence in a non-tty deploy context; image exists is
the boolean primitive and cannot misreport. Absent-image behavior
(pull from ghcr) is unchanged.
podman's reference parser rejects mixed-case repository refs before any
network I/O ('repository name must be uppercase'); upstream's own install
docs pull lowercase (ghcr resolves case-insensitively). The first deploy
of this chain exit-2'd on that parse error — the image was not yet
pulled, so the guarded vhosts 502 until this deploy lands.
The first deploy run dispatched caddy (hot reload, no restart) but exit-2'd
on the inventory guard before deploy_anubis could run: deploy.sh's
valid_service was extended but the remote runner's per-target inventory
was not. Guarded vhosts 502 until this lands.
Bots and AI scrapers hit the SSO-fronted vhosts and eat Authentik auth
work for 403s. Anubis verifies clients first: DENY for known AI scraper
UAs, JS challenge for everyone else; verified traffic resumes the normal
Authentik forward_auth chain on Caddy's internal listener (:8016).
Chain: client -> Caddy (TLS) -> Anubis :8923 (verify) -> Caddy :8016
(auth + routing resumes; Host survives both hops) -> app. Reverse-proxy
through Anubis, not forward_auth: at the pinned tag a CHALLENGE answers
HTTP 200 + HTML, and forward_auth treats 200 as "authenticated" — the
page would pass through unchallenged. Exempt by construction (never
routed through Anubis): plex (plain HTTP legacy), auth.* (IdP login
must be reachable ungated), gitea (git smart-HTTP clients cannot solve
JS challenges), and every app with its own login.
Pinned ghcr.io/TecharoHQ/anubis:v1.13.0 — the tag was source-verified
before pinning: at this tag the policy parser is JSON-only with a
minimal bots[] schema (no imports, no CEL expressions), flags are the
config surface via flagenv (TARGET/POLICY_FNAME map to --target/
--policy-fname), and the image's exec-form HEALTHCHECK is inherited by
the quadlet unit (dodges the v1 generator's HealthCmd space-split quirk).
Rollback nuance: stopping anubis 502s everything behind the chain (the
public listeners proxy through it); a true rollback is reverting the
Caddyfile and redeploying, not just systemctl --user stop anubis.
Verified live on dankclaw: the podman 5.x v1 quadlet generator splits
HealthCmd values on spaces into the container healthcheck argv array, so
quoted probes (sh -c "pg_isready ...") arrive mangled and every probe
exits 127 — postgres read 'unhealthy' with the database actually up.
Probe forms now use only space-free argv tokens: pg_isready passes its
args directly; the server probe runs manage.py check --database default
(verified exit-0 standalone in the image).
Probed the live generator on dankclaw: podman 5.x v1 hard-fails unit
generation on dash-prefixed options (--shm-size killed authentik-server's
unit) and rejects Command= entirely; Exec= is the accepted role selector.
Drop --shm-size (uwsgi uses unix sockets under /run, not /dev/shm; default
64m shm is fine) and drop systemctl --user enable — generated units are
transient and cannot be enabled; WantedBy=default.target in each unit file
is what makes them boot-persistent on this generator.
The compose-era authentik trio died at every boot (rootless podman does not
auto-start bare containers). Convert the trio to Quadlet units installed by
deploy-remote.sh into ~/.config/containers/systemd/, enabled + WantedBy=
default.target:
- 3 unit files under dankclaw/authentik/quadlet/ (postgres -> server -> worker
ordering via Wants=/After=; secrets stay in the git-ignored host .env,
expanded by systemd at start time exactly like the gitea units)
- deploy_authentik() handler: install units, daemon-reload, retire stale
compose-era containers only while units are inactive, enable once, then
start in dependency order with podman-health gates between stages
- compose.yaml: authentik service definitions retired to a comment pointing at
the quadlet dir (same pattern as searxng/open-webui/gitea/caddy)
- tests/authentik.bats rewritten from compose assertions to unit-file
contract assertions (images, Exec roles, ordering, loopback-only publish,
pg_isready probe, secret hygiene, dispatch routing)
All in-repo path references (AGENTS.md, README, old scripts, report artifacts,
test code paths) rewritten in the same commit; focused-test map extended for
.gitignore, old/scripts/*, dankclaw/comfyui/*, and report/* per this repo's
own convention ('add one before committing'). Obsidian wikilinks are
title-based and unaffected.
Both the EasiiX Q8_0 and agentionai ROCmFP4-FAST MTP draft heads fail
with 'missing tensor blk.0.hc_attn_norm.weight' against the STRIX_LEAN
Uncensored model. The qwen4exp binary expects this hyper-connection
attn norm tensor which neither draft GGUF carries. Using ngram-mod
only until a compatible MTP head is found.
The EasiiX Q8_0 draft was incompatible with STRIX_LEAN (missing
blk.0.hc_attn_norm.weight). Switched to agentionai/Qwen3.8-Flash-Next-
MTP-ROCmFP4-FAST-GGUF (2.28 GiB, ROCmFP4-FAST quant matched to target).
n_max 3 per the HF card (n_max 4 measured worse). SPEC_TYPE=draft-mtp,ngram-mod.
The script runs ON Dankclaw but was calling ssh() to SSH back to
dankclaw for service management. Nested SSH fails (no key auth for
localhost). Replaced ssh() with local() using subprocess.run(shell=True)
for systemctl and sed commands.
The env file had the old STRIX paths (directory doesn't exist on disk).
Updated to the actual running STRIX_LEAN (Uncensored) variant:
- MODEL_DIR → kingjones-Qwen3.8-Flash-Next-Uncensored-ROCmFP4-STRIX_LEAN
- MODEL_FILE → 3-shard Q4_0 (00001-of-00003)
- MMPROJ_PATH → mmproj-Qwen3.8-Flash-Next-Uncensored-BF16.gguf
- serve-frank-large.sh: add --mmproj $MMPROJ_PATH
Reconciles the uncommitted local changes that were on Dankclaw.
serve-frank-large.sh: only pass --model-draft when SPEC_TYPE contains
'draft-mtp'. ngram-only configs (ngram-mod, ngram-simple, etc.) do not
need a draft model file — loading one wastes VRAM and may error.
bench-spec-decode.py: new benchmark script. Sweeps 4 configs (none,
ngram-mod, draft-mtp, draft-mtp+ngram-mod), generates 16k tokens per
config at temp 0 from a minimal prompt. Uses HTTP API (llama-bench
cannot target --spec-type). Follows benchmarking policy: stop+mask
the production service, bench, unmask+restart, health check.
podman's quadlet generator creates the default.target.wants symlink from
the unit's [Install] section at every user-manager start (is-enabled
reports 'generated' for all healthy quadlets, pihole included). An
explicit 'systemctl --user enable caddy.service' fails with 'transient or
generated' and aborted the deploy after the unit was already installed.
systemd-sysctl applies /etc/sysctl.conf AFTER all of sysctl.d/, so the
stale ip_unprivileged_port_start=80 line there beat every drop-in and kept
the runtime value at 80 after the first deploy pass. deploy_pihole now
comments out an active line in /etc/sysctl.conf (idempotent) and generalizes
the sysctl.d conflict scan to any file setting a value above 53.
Caddy left compose.yaml for a quadlet unit (caddy.service): joins
dankclaw_default for container-name upstreams, Pull=never for the local-only
image, Porkbun credentials via /home/dank/Containers/caddy/env written by
deploy from .env, bare-IP-catch-all health check. deploy_caddy installs,
enables, and starts the unit after retiring the bare container (state is in
host volumes).
Pi-hole: the repo now owns the sysctl drop-in
(dankclaw/pihole/sysctl/99-dnsmasq.conf, ip_unprivileged_port_start=53);
deploy_pihole installs it to /etc/sysctl.d/, stages conflicting later-sorted
drop-ins in the home trash, and applies 'sysctl --system' (the boot code
path). This fixes the 2026-08-26 crash loop (port-53 bind denied, restart
counter 1645) caused by a second drop-in resetting the value to 80.
Tests: new caddy_quadlet.bats seam; caddyfile/deployment_flow assertions
repointed from the removed compose service to the quadlet; pihole.bats pins
the sysctl drop-in and its deployment; test-changed.sh routes dankclaw/caddy/
and the pihole subtree to their seams. Docs: Caddy, Pi-hole, Dankclaw Stack,
work log.
A bare compose container dies silently at reboot (rootless Podman has no
boot-time auto-start; 'restart: unless-stopped' does not survive shutdown's
explicit stop). Rule established by the 2026-08-26 Dankclaw reboot, which
left Caddy down and Pi-hole crash-looping on a sysctl conflict.
--ctx-size is the total pool divided among --parallel slots, so with
PARALLEL=3 the 160k target must be a 491520 pool (163840/slot) to
actually deliver 160k context per slot. Env, serve default, bats updated.
Reconfigure to 3 parallel 160000-ctx slots (was 1 slot). KV cache is
now asymmetric: K at q8_0, V at q5_0, both with Hadamard. The 3 slots
triple the KV, so expert blocks 27-34 are pinned back to CPU (freed
~8.7 GiB) leaving output-adjacent 35-40 on CUDA. Q8_0 model,
split-mode none, spec disabled (ik limits spec to 1 slot). AGENTS.md:
drop the scratch-port rule (test/bench processes cannot share the
single GPU with the running service anyway). Env, serve script, bats,
docs updated.
Switch to CRACK Q8_0 model (37.8 GB, mmproj f16). Servt as a single
160000-ctx slot (PARALLEL=1); speculative decoding stays disabled. KV
cache is q6_0 + Hadamard. iq5_k is not a valid KV-cache type in ik_llama
(cache whitelist excludes iq5_*); q8_KV reproduces the ggml.c:5427 view
assert even at PARALLEL=1/no-spec, so q6_0 is the verified working 6-bit
choice. Env, serve script, tests, and KB updated. Deployment was left
down after the isolation test; this commit redeploys it with the
verified Q8_0/160k/q6_0 config.
Fused MoE (fused_moe / fused_up_gate / fused_mmad) is on by default in
ik_llama - no flag needed; only -no-fmoe disables it. Slide two more
expert blocks (28,29) from CPU onto the RTX 3090 (~1.26 GiB) for the
fused-MoE run, leaving ~1.6 GiB headroom. Update override regex, test,
and KB (including a note that the headroom is now tighter than the
2-3 GiB ideal).
--jinja: ik_llama only renders a custom --chat-template-file as real Jinja
when --jinja is passed (inputs.use_jinja ? apply_jinja : non-jinja
fallback). Without it the Qwen-Sharp template's thinking/image/tool
formatting is silently dropped. Add USE_JINJA=1 -> --jinja.
Experts: the RTX 3090 had ~5.4 GiB headroom; pull expert blocks 30,32,33,34
back to CUDA (each block ~630 MiB) from CPU for lower latency. Update the
OVERRIDE_EXPERT regex, its test, and the KB with the headroom-check note.
ik_llama limits speculative decoding to a single server slot (-np 1) and
hard-exits at load if a spec stage is configured with --parallel > 1:
"Speculative decoding is currently limited to a single server slot
(-np 1)." This service keeps PARALLEL=3 (one slot per user), so set
SPEC_TYPE=none to disable speculation. It is a low-parameter MoE (3B
active of 35B), so throughput is fine without it. MTP stays disabled.
KB and tests updated.
The ik_llama fast q8_KV cache type crashes at model init on this
QWEN3MOE/MTP + multimodal arch at commit b166e269 with a ggml_view size
assert (ggml.c:5427) right after the output buffer is created, looping
the service. Empirically isolated by toggling cache type: q8_0 and f16
load and serve (smoke-tested correct output), q8_KV asserts. Switch to
q8_0 + Hadamard (K/V), which still gives the 8-bit cache + Hadamard
quality win. KB updated with the crash lesson.
--split-mode graph requires >= 2 devices in ik_llama.cpp and errors hard
at model load on a single RTX 3090 ("It is not possible to use split mode
'graph' with less than 2 devices"), looping the service. Set SPLIT_MODE
=none (the default), which works with the --override-tensor CPU expert
offload. KB updated with the two-device requirement so we do not assume
graph degrades gracefully on one GPU.
Switch frank-default-small on Dankserver from upstream llama.cpp (b10121)
to ikawrakow/ik_llama.cpp (b166e269, built with the staged pip cu13
toolkit). The model (Ornith-1.5-35B-A3B CRACK Q6_K) is unchanged and the
old binary is kept for rollback.
Runtime changes:
- KV cache quantized to q8_KV with --k-cache-hadamard/--v-cache-hadamard
- --split-mode graph (single-GPU no-op, retained per operator pref)
- migrate ngram speculation to canonical
--spec-type ngram-mod:n_min=48,n_max=64,ngram_size_n=24
(legacy --spec-ngram-mod-* are rejected in ik main)
- N_GPU_LAYERS 999 (ik -ngl rejects the literal "all"); experts stay
pinned to CPU via --override-tensor
Build: GGML_CUDA_FA_ALL_QUANTS=ON + GGML_IQK_FA_ALL_QUANTS=ON mandatory
for the quantized KV cache; recipe + reasoning baked into
docs/LLM/ikllama-ornith.md. Tests and both indices updated.
Caddy v2.11.4's caddyfile adapter rejects 'tls { insecure_skip_verify }'
inside transport http (unrecognized subdirective '{') and also rejects
the handler-level form. The working form, verified live against PMS with
a throwaway caddy (200 OK), is the bare token:
reverse_proxy 192.168.0.235:32400 {
transport http {
tls_insecure_skip_verify
}
}
PMS 1.43.x on :32400 accepts, reads, and silently closes every non-TLS
connection from a non-loopback source (loopback exempt). The plain-HTTP
reverse_proxy therefore got EOF and Caddy answered an empty-body 502.
Both Plex vhosts now dial the upstream over TLS with
insecure_skip_verify (PMS self-signed cert, pinned LAN hop only;
client-facing Let's Encrypt TLS unchanged).
dankware carries two +DefaultRoute links (enp47s0 eth, wlan0 wifi) that
both receive the same DHCP DNS list. systemd-resolved picked a different
'current' resolver per link, so flat-name resolution flapped between the
LAN IP (192.168.0.143 via wlan0/Pi-hole) and the WAN IP (Comcast v6 via
enp47s0).
Pin 192.168.0.143 as the static IPv4 resolver on both NetworkManager
connections (ipv4.method=auto keeps the DHCP address/gateway/route,
ignore-auto-dns=yes uses only Pi-hole, ignore-auto-routes=no keeps the
DHCP routes), then restart systemd-resolved. resolved.conf's own DNS= is
a per-link fallback, so the link must be pinned.
Flat names now resolve to 192.168.0.143 on every query; apex/unknown
subdomains still fall through to Porkbun (71.227.227.172).
The v6 restore commits dropped --chat-template-file and
--chat-template-kwargs while the artifact check still validated the
template file, so the Sharp template was checked but never applied.
Re-add both flags (with --jinja) so reasoning is emitted in DeepSeek
format and preserve_thinking/preserve_reasoning are honoured.
Reformat the script one-flag-per-line with long flag aliases
(--gpu-layers, --ctx-size, --ubatch-size, --cache-type-k/v) like
serve-frank-small, and update the 2 frank_large_config assertions
that grep the old short-flag forms (-ctk/-ctv, -ub).
- scripts/serve-frank-large.sh
- tests/frank_large_config.bats
The pre-commit hook runs scripts/test-changed.sh, which requires a
focused-test mapping for every staged path. The quadlets were moved into
dankclaw/<svc>/quadlet/ but the mapping still only matched top-level
dankclaw/*.container, so any nested quadlet or the pihole addn-hosts/env
data files fell through to the catch-all 'die' and blocked commit.
Add the nested-quadlet mappings (pihole-specific first, then the
dankclaw/*/quadlet/* fallback) plus the pihole data-file paths.
Frank