Commit Graph
133 Commits
Author SHA1 Message Date
Frank 4fca739bf8 frank-medium: deploy two single-GPU replicas — TP=2 is blocked by PCIe atomics
The radiance MXFP4 stack cannot run TP=2 on this host. RCCL's collective
kernels carry a hidden hostcall-buffer parameter, and ROCr refuses to
dispatch them on a device without PCIe atomics: the AQL dispatch fails with
hipErrorIllegalState ("the operation cannot be performed in the present
state"), which is what every two-rank collective hit. The second R9700 sits
behind the chipset bridges 05:00.0/06:09.0, which advertise
`AtomicOpsCap: Routing-` — a read-only capability, so no software can enable
atomic routing on that path and the driver's
pci_enable_atomic_ops_to_root() fails for that card. The first card, on CPU
root port 00:01.1, has atomics enabled.

Ruled out: kernel version (7.0.0-28/-30/-31 identical, after a full upgrade),
iommu=pt, P2P hardware (45.3 GB/s between the cards), the shipped librccl (a
from-source rebuild reproduces the failure at the same call path), vLLM and
pynccl (a bare two-rank torch.distributed allreduce fails the same way), all
NCCL transport knobs, and single-rank tests (NCCL short-circuits at
world_size=1, so one-card success proves nothing).

Deployment: three rootless Quadlets — frank-medium-a (:7771, card 1),
frank-medium-b (:7772, card 2), both TP=1 at 114688 context (262144 and
131072 do not fit one card's KV) with vLLM-profiled KV — and
frank-medium-proxy (:7770), a round-robin that preserves the fleet contract
of one endpoint and one served model name. frank-medium.container stays as
the TP=2 target for a host whose second card is atomic-capable; the deploy
handler does not install it and trashes any stale copy.

Measured through the proxy (vllm bench serve, random dataset, --ignore-eos):
conc 1 42.8 tok/s, conc 4 147.6 tok/s, conc 8 230.8 tok/s, 32k prefill at
~2.8k tok/s per card, TPOT p50 23.6-28.8 ms.

Also: healthcheck reads PORT so one probe serves all three units, the hermes
example and the harness models.yml drop to the real 114688 context, and the
runbook records the full evidence chain.
2026-09-16 03:47:11 -07:00
Frank 0c1e10a5ef frank-large: retire the halogen quadlet test seam (ik_llama.cpp cutover) 2026-09-16 00:38:34 -07:00
Frank 0022ba96ad frank-large: mmap the CPU-overridden weights (no CUDA_Host pinning) 2026-09-15 23:58:34 -07:00
Frank fe3fc62c41 frank-large: swap halogen Quadlet for ik_llama.cpp (3090 + Strix Halo CPU path) 2026-09-15 23:20:19 -07:00
Frank 6b647c1558 frank-medium: swap ROCmFPX llama.cpp runtime for vllm-radiance MXFP4
Rootless Quadlet on dankserver serving
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (the radiance stack's
designated alternate target; ships its fp8 MTP head, no rewrite step) with
the DFlash2-FP8 drafter at SPEC=7 on the 2x R9700s (TP=2, host network,
:7770, alias frank-medium unchanged).

- dankserver/frank-medium/: quadlet unit, container entrypoint (fork of
  serve-mxfp4.sh's patch block at pinned clone commit 97353293), healthcheck
- dispatcher retires the root llama.cpp unit, re-tags the registry image to
  the localhost alias, installs the quadlet, waits on /health
- planner maps dankserver/frank-medium/* to frank-medium; scripts-era
  mappings removed
- ROCmFPX-era env/serve/unit/download scripts staged in scripts/.AGENTTRASH
  with .rocmfpx suffix; seams updated (frank_medium_quadlet.bats new)
2026-09-15 20:59:55 -07:00
Frank 6085a99dd8 frank-medium: serve the new R9700s, retire frank-small
The RTX 3090 left Dankserver; two AMD Radeon AI PRO R9700 cards (Navi 48,
gfx1201, 32,624 MiB each) replaced it. FreeToken/CUDA cannot run there at all,
so the `frank-small` service is retired and `frank-medium` takes the endpoint.

New service: charlie12345/ROCmFPX llama.cpp (built for gfx1201 against the
fork's self-contained ROCm 7.14 toolchain, so the host's stale /opt/rocm-6.3.3
never participates) serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF --
a Q8_0_ROCMFPX_AGENT (ftype 115) checkpoint that stock llama.cpp cannot load.
Placeholder config: --split-mode tensor across both cards, 131072 context,
1 slot, f16 KV, MTP draft head on ROCm0.

deploy-remote.sh grows retire_root_llm_small(): the CUDA-era unit is stopped,
disabled and moved to ~/.AGENTTRASH/20260915-frank-small-retire/host-units
before frank-medium starts, because `systemctl disable` has been observed to
delete the unit file itself on this host (2026-08-26 incident).

The retired frank-small repo files (env, unit, serve script) move to
scripts/.AGENTTRASH/, and the planner, dispatcher, service allow-list and the
deployment test seams now name frank-medium.
2026-09-15 18:53:20 -07:00
Frank 57dbcc5459 e2e: add Marreta homepage-link regression test
16-marreta.spec.ts (standard project) asserts the user homepage's Marreta
open link actually reaches the Marreta app through the full guard chain
(Anubis PoW -> Authentik SSO -> app), and explicitly NOT the authentik
login flow. Guards the 2026-09-12 incident where marreta-proxy was created
without a linked Application, so the embedded outpost never loaded it and
the homepage link parked on a login wall.

Wires spec 16 into the standard project testMatch. Verified passing live.
2026-09-12 13:31:00 -07:00
Frank 73e46a1d85 marreta: content-cleaning proxy on Dankclaw, gluetun netns, guard-chained
Deploy manualdousuario/marreta (Laravel / serversideup php:8.4-fpm-nginx)
as a rootless Quadlet pair (marreta + marreta-browser/Lightpanda), both
joining gluetun's netns so all fetch egress exits the same ProtonVPN
tunnel as SearXNG.

- marreta relocated to NGINX_HTTP_PORT=8085 (netns already has searxng on
  8080); gluetun publishes 8082:8085; Caddy dials 192.168.0.143:8082.
- Guard chain: marreta.franksdank.cloud -> Anubis :8923 -> Authentik ->
  post-auth :8017 @marreta -> 8082. Access = any authenticated user.
- Secrets seeded to /home/dank/Containers/marreta/env (0600) from
  MARRETA_APP_KEY / MARRETA_ADMIN_EMAIL / MARRETA_ADMIN_PASSWORD.
- deploy-plan.py: marreta joins the gluetun netns family.
- New seam tests/marreta_quadlet.bats (17 contracts); scoping test updated
  for the widened netns family. Doc: Dankclaw/Marreta.md + index + log.
2026-09-12 10:09:12 -07:00
Frank 25c93984ad fix(build): build the loki/alloy/comfyui images in docker format
The healthcheck added to the loki image was discarded by the build:
'HEALTHCHECK is not supported for OCI image format and will be ignored'.
podman builds OCI by default, and OCI has no HEALTHCHECK field, so the
container came back with no health at all — worse than the failing check it
replaced, because nothing reports the loss.

The build now passes --format docker, and the seam pins it so a future edit
cannot quietly reintroduce an image with no health field.
2026-09-11 16:02:47 -07:00
Frank b96359e628 fix(observability): give loki an exec-form healthcheck in its image
The loki unit came up 'unhealthy' after the quadlet cutover. Cause: a unit's
HealthCmd reaches podman as CMD-SHELL, i.e. /bin/sh -c, and the loki image is
distroless — no shell exists in it. The busybox command itself works (probed
by hand: /ready answers 'ready'); it was the shell wrapper that could not run.
podman does not accept an exec-form command through --health-cmd either: it
stores a JSON array as one literal argument.

The command therefore moves into dankclaw/loki/Containerfile as a
HEALTHCHECK ... CMD [...] exec form, which is what that image exists for — it
already ships the static busybox solely so a healthcheck has something to run.
podman reads the image's healthcheck when the unit declares none.
2026-09-11 16:01:21 -07:00
Frank fe4eeaa03a fix(observability): seed grafana's OAuth secret into a per-service env file
The first deploy of the grafana unit failed: 'stat /root/.config: permission
denied', exit 1, restart loop. Cause — the repo-root .env that the compose
service used to pick up is a gluetun env file, and it carries HOME=/root (and
the WireGuard keys). Pointing the unit's [Service] EnvironmentFile at it sets
HOME for the service, so podman looks for its configuration under /root.

grafana now reads /home/dank/Containers/grafana/env, the per-service file the
deployer seeds from dankclaw/.env with a single-variable filter — the same
shape caddy, pihole, copyparty, romm and seerr use. The seed is validated, so
a missing secret fails the deploy loudly instead of starting a container whose
OAuth login cannot complete. The value was copied to dankclaw/.env, which is
where dankclaw/.env.example already documented it.
2026-09-11 15:59:15 -07:00
Frank dab81b91a3 feat(observability): make the five Dankclaw observability containers quadlets
prometheus, grafana, loki, alloy and node-exporter ran only from
dankclaw/compose.yaml. Rootless podman does not restart bare containers, so
they did not survive the 2026-09-11 reboot for the amd_iommu=off change and
had to be restored by hand — the same gap that had already cost the fleet
Caddy and Pi-hole, which is why the boot-persistence rule exists.

Each is now a rootless Quadlet unit under dankclaw/observability/quadlet/,
installed and dispatched by deploy_observability in deploy-remote.sh, with
the compose definitions retired so only one owner exists per container name.
Images, mounts, arguments and container names are unchanged, so Caddy's
proxies and Prometheus' own scrape config keep resolving.

Two posture changes, both about the unauthenticated surface:

* The three HTTP UIs published on 0.0.0.0 with no authentication of any kind
  (Prometheus even with the lifecycle endpoint enabled). They now publish on
  127.0.0.1: the browser path is the guarded franksdank.cloud vhost, Caddy
  reaches them by name over dankclaw_default, and the host-side homepage
  status timer probes loopback. Nothing legitimately used the LAN port.
* node-exporter bound :9100 on every interface; it now binds exactly its two
  consumers, loopback (homepage timer) and the LAN address (Prometheus, which
  runs in a container and cannot reach the host's loopback).

alloy gains a reachable health endpoint: its UI listens on the container's
own loopback and alloy 1.x refuses a config block for it, so the compose
service had nothing to probe — the homepage status card for alloy read 'down'
continuously. The unit moves the UI with a command-line flag and publishes
127.0.0.1:12345.

grafana keeps its Authentik OAuth client secret without receiving the whole
.env: systemd expands the value from a [Service] EnvironmentFile, which never
reaches the container.
2026-09-11 15:56:39 -07:00
Frank 301914b1c0 feat(llm): swap frank-large to halogen-flash-server, retire frank-npu
frank-large moves from the kingjones30/ROCmFPX llama.cpp build to
peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b
.hgn 115.55 GiB + quality overlay). Upstream measures this engine at
roughly 4x the prefill throughput of the llama.cpp family on the same
gfx1151 silicon, and prefill dominates any prompt with real context.

Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/),
engine on container loopback :8730 (never published), OpenAI front-end
:8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client
renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB,
conservative for a shared 121 GiB host), 4 slots, host reserve 24,
vision tower on, weights pinned, all other flags at vendor defaults.
Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable
/home/dank/models/halogen.

The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/
rather than deleted; the STRIX payload and the ROCmFPX build stay on
disk as the rollback target. deploy_llm_large becomes a user-quadlet
handler: it retires the old ROOT unit (moving its checkout symlink into
the home trash, never rm) and gates the first start on the download
rather than reporting a failure for a fetch that is working exactly as
designed.

frank-npu is retired in the same change. The kernel cmdline moves from
iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured,
and that takes the XDNA2 NPU with it: without PASID/SVA the driver
cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are
repointed (Hermes vision -> frank-large, Hermes title generation ->
frank-small, harness vision -> frank-large, tiny -> frank-small, task ->
frank-large:medium) and its provider blocks are removed from both the
harness config and Hermes's live config. Audio input leaves the fleet
with no replacement; image input survives on halogen's vision tower.

Docs: new halogen-flash-server runbook, new IOMMU decision record, and
frank-npu rewritten as a retirement record. AGENTS.md topology, runtime
sections, benchmarking policy (frank-large is a user unit now) and the
harness model-config section are corrected.
2026-09-11 14:48:28 -07:00
Frank b6f5b2bcf5 test(e2e): pin unauthenticated wget-style denial for copyparty and homepage
Wget-style (no-JS) clients must never receive application bytes from guarded
services: copyparty serves authenticated traffic with anonymous read perms, so
the guard chain is the only boundary between an unauthenticated client and the
file bytes. Assertions target leakage markers in app code (Copyparty UI title,
homepage panel grid, services.json JSON shape), not gate internals, so any
guard-route regression fails the suite while any gate implementation passes.

Also route 15-*.spec.ts into the guest project (empty storage state).
2026-09-04 19:34:59 -07:00
Frank df4339a82a docs(arr): fleet consolidated on FrankBook-Pro, VPN egress proven; E2E counts derived from registry 2026-09-04 13:17:39 -07:00
Frank 4c7465d00d arr fleet: FrankBook-Pro is the sole deployment, all egress VPN-gated
FrankBook-Pro becomes the only *arr/qBittorrent deployment. Every fleet
container (qB + five arrs) now shares gluetun's netns, so all egress —
torrent, indexer APIs, metadata lookups — exits the WireGuard tunnel by
construction (AllowedIPs=0.0.0.0/0); the arrs' own ports publish on the
gluetun unit, none on the arr units themselves.

- dankserver/compose.yaml: remove the six arr services (the Aug 20 compose
  copy — never healthy, superseded; config dirs trashed on the host)
- dankclaw/Caddyfile + shared/homepage/services.json: arr vhosts and
  probes repoint 192.168.0.235 -> 192.168.0.112 (FrankBook-Pro)
- frankbook-pro units: five arrs join gluetun netns (Wants/After/Network);
  gluetun publishes 9696/8989/7878/8686/8787 alongside 8080/6881
- tests/frankbook_quadlet.bats: new VPN egress seam — every arr+qB unit
  must share gluetun's netns and publish no port; gluetun must publish the
  whole fleet's ports
- deploy-plan.py: gluetun changes carry the whole netns family (a gluetun
  recreation kills all six joiners); dankserver compose set loses the arrs
- AGENTS.md: deployment is complete only when unit tests AND the Playwright
  E2E suite both pass
2026-09-04 12:54:28 -07:00
Frank b563053966 test(e2e): accept IdP redirect as a valid gate in the direct-nav guard spec
Live run against the fleet showed the outpost redirects direct navigation to
auth.franksdank.cloud with the guarded host preserved as redirect_uri, rather
than rendering the flow UI in place. The security property under test is that
an unauthenticated visitor never sees app content; assert that shape (uidField
visible + no leak markers + gate is either in-place or a redirect carrying the
return path) instead of hostname preservation.
2026-09-03 17:00:34 -07:00
Frank 5a083eee6c test(e2e): prove guard chain blocks direct navigation and admins keep admin reach
14-guard.spec.ts: every authentik-guarded vhost (www, search, romm, files,
seerr, grafana, prometheus, loki) must challenge an unauthenticated direct
navigation; plex is asserted exempt (PMS answers directly, no IdP form).
11-admin.spec.ts: admin direct-nav to grafana/prometheus/loki passes through
with host preserved and no uidField challenge.
2026-09-03 16:16:11 -07:00
Frank 69f87c8483 feat(llm): rename inference services dropping 'default' from names
frank-default-large -> frank-large (Dankclaw :7770), frank-default-small ->
frank-small (Dankserver :7770), frank-default-sightsound -> frank-npu
(Dankclaw :7771). Units, env files, serve scripts, deploy-plan mappings,
deploy-remote dispatch (stop/disable old units + relink + enable new), and the
test seams move together. Retired frank-default-npu.* artifacts moved to
scripts/.AGENTTRASH/ (tracked-in-history; ignored on disk).

Served aliases become frank-large / frank-small via --alias; the NPU endpoint
serves the whole FLM registry, so its rename changes no served tags.
2026-09-03 16:15:43 -07:00
Frank 58aa084d9c quadlet: drop the userns identity model — rootless cannot map uid 997
Evidence from the failed first cutover: podman top shows container root
maps to dank (1000) under UserNS=host rootless (subuid range 100000:65536
cannot express 997), so PUID=997 fell back to root and PMS crash-looped on
write access to its 997:984-owned data root. Fleet model instead: PUID=0/
PGID=0 (container root = dank, same as jellyfin/copyparty data roots) and
deploy_plex chowns /var/lib/plexmediaserver to 1000:1000 inside the
first-cutover block. Zero bytes moved; rollback regains one chown -R
997:984 step, documented in Plex.md. Seam test pins PUID=0/PGID=0 and the
absence of any userns key.
2026-09-01 22:46:00 -07:00
Frank 1939135e4c quadlet: UsernsMode= is compose syntax — use the v1 key UserNS=host
podman 5.7 podman-user-generator on dankserver hard-fails the whole unit on
the unknown key ('unsupported key UsernsMode in group Container'), so the
first cutover attempt masked the native unit and then found no plex.service
at all. Swap both units to UserNS=host (same --userns=host flag), update the
ownership-contract seam test to pin the exact key, and correct both docs.
Pin the seam: the v1 generator rejects unknown Container keys silently at
deploy distance — the test now pins UserNS=, not the misspelled alias.
2026-09-01 22:36:43 -07:00
Frank 93f16f68be deployer: route plex and tautulli as dankserver quadlet services
- deploy_plex(): install unit, stop+mask native plexmediaserver.service on
  first cutover (mask, never disable), gate on /identity for 180s
- deploy_tautulli(): install unit, stop+disable snap on first cutover,
  seed config.ini with PMS_Web_Host=plex only if absent
- allow-lists (deploy.sh valid_service, deploy-remote.sh service_is_allowed),
  planner routing (SERVICE_ORDER, DANKSERVER_SERVICES, quadlet_services),
  test-changed cutover case, quadlet seam enumeration + new
  userns/PUID=997/PGID=984 ownership-contract test
2026-09-01 22:31:14 -07:00
Frank 54239294b7 authentik: require flows + application on proxy providers; harden SSO e2e
The loki/grafana/seerr proxy providers were script-provisioned with no
flows and no Application: null flows made the embedded outpost's Rust
deserializer reject the whole config (all guarded vhosts down after any
outpost reload), and the missing apps bounced post-login to authentik's
user UI instead of the app. Backfilled both live; documented the two
provisioning rules and updated the provider inventory. SSO e2e now
asserts the app landing (pathname leaves /if/, no IdP markers in body)
instead of only the hostname.
2026-09-01 18:55:44 -07:00
Frank 4e1c8a873f e2e: harden akLogin against Lit stage re-render; settle per hop in crawl/sso 2026-09-01 17:09:40 -07:00
Frank 912c0edeff e2e: budget user/admin title asserts to 60s for cold Anubis PoW 2026-09-01 17:09:17 -07:00
Frank dfd9c8d1c0 e2e: Playwright suite for homepage + guard chain; RBAC gate on /admin; fix stale service links 2026-09-01 16:42:33 -07:00
Frank f4c9b4f381 seerr: quadlet on dankclaw + guard-chain vhost + deployer wiring
Unit: ghcr.io/seerr-team/seerr:v3.4.1, loopback-only 5055 publish
(homepage probe; users go through Anubis->Authentik->caddy:8017->seerr),
Containers data root, no env secrets. Guard chain: seerr.franksdank.cloud
vhost behind anubis:8923 with post-auth handler on 8017; addn-hosts record;
homepage widget; deploy-remote/deploy-plan/deploy.sh routing; seam test.
2026-09-01 14:40:10 -07:00
Frank 949f68e2b5 frankbook-pro: *arr stack + qbittorrent quadlets inside podman machine
Seven units installed INSIDE the Mac VM (/home/core/.config/containers/
systemd) by a new deploy_frankbook_stack handler: gluetun (ProtonVPN
WireGuard, env-file credentials, publishes qbit's 8080/6881), qbittorrent
(Network=gluetun.container, verbatim /Volumes data roots, first-boot
config seed with fresh machine-local WebUI password and the Finished
Torrents category), and prowlarr/sonarr/radarr/lidarr/readarr (grab-only,
config migrated from dankserver at provisioning, readarr pinned to
amd64-1.0.980). deploy.sh routes the new target through the ssh-config
alias with REMOTE_SHELL=/usr/local/bin/bash (macOS bash 3.2 lacks
associative arrays); the planner learns FRANKBOOK_SERVICES and the
gluetun->qbittorrent netns pairing; new seam test pins the contract.
2026-09-01 14:10:23 -07:00
Frank a14a70a404 dankserver: copyparty PublishPort=3923 + seam pinning host-served ports
The cutover unit joined dankserver_appnet but dropped the compose-era
0.0.0.0:3923 mapping (live inspect: bridge, 3923 published), so
copyparty listened inside the netns while the host refused. New seam
asserts every host-facing port matches the pre-cutover PortBindings:
romm 8085->8080, jellyfin 8096, calibre-web 8083, copyparty 3923.
2026-09-01 13:53:10 -07:00
Frank 06de2b4cbf dankserver: jellyfin GPU device via AddDevice=, seam guard for Device=
The podman 5.x v1 quadlet generator rejected the whole jellyfin unit on
the unsupported Device= key (generator log: unsupported key 'Device' in
group 'Container'), so no jellyfin.service was generated and the cutover
restart failed with 'Unit not found' after the data root had already
migrated cleanly. gluetun proves AddDevice= is the house key. New seam
test denies Device= across all six dankserver units and pins the
AddDevice=/dev/dri replacement.
2026-09-01 13:49:45 -07:00
Frank b206f477ad dankserver: quadlet units + deploy handlers for romm trio, jellyfin, calibre-web, copyparty
Cutover of the six surviving dankserver services from compose-era bare
containers to boot-persistent Quadlet units (WantedBy=default.target; podman
5.x v1 generator honors it at user-manager start).

Units are faithful to live inspect ground truth (image pins = running tags,
not compose inventory):
- romm trio shares one data root: /home/dank/Containers/romm mounted whole at
  /romm; mariadb datadir and redis data dir live inside it. Aliases mariadb/
  redis preserved on dankserver_appnet via Network=...:alias=. Secrets stay in
  the host-local env file (EnvironmentFile + ${VAR} expansion in the generated
  ExecStart, gitea precedent). copyparty passes its admin account via
  Exec=-a ${COPAY_ACCOUNT}:${COPAY_PASSWORD} (Exec= not Command=: the v1
  generator hard-fails on Command=); entropy stays rw only there.
- jellyfin keeps every creation-time env as Environment= lines (zero drift);
  Device=/dev/dri + GroupAdd=keep-groups preserve hardware transcoding.
- calibre-web pins the exact running image tag (id 986277563e76 =
  0.6.27-ls397), pasta network preserved.

Deployer: first-cutover handlers retire the old container, rescue unshadowed
anonymous-volume content (podman volume export + tar --skip-old-files), mv
data roots on the same fs, then start units in dependency order with health
gates. Re-deploys restart only the selected unit. deploy-plan.py maps compose
keys mariadb/redis to the new unit names; allowlists updated. New seam test
tests/dankserver_quadlet.bats pins boot-persistence + v1-generator contract;
test-changed.sh maps the new dirs to it. Full suite green (209 tests).
2026-09-01 04:14:08 -07:00
Frank c6d404ffde AGENTS.md: External Exposure Policy (MANDATORY); comfyui_rocm seam retargeted
Every Caddy-proxied service is now either behind the auth chain or
explicitly internal-only, with the provider-first flip procedure codified.
The comfyui GPU-access seam test now asserts the deployed contract
(group_add keep-groups + udev MODE 0666 on kfd/renderD*) instead of the
pre-fix 'video' group assumption.
2026-08-30 23:26:18 -07:00
Frank 29f9a86884 caddy+authentik tests: uniform Host-preserving :8016 proxy through embedded outpost
The redirect-to-flow design with an absolute cross-host next= was rejected by
the flow's open-redirect guard at completion ("Request has been denied:
Invalid next URL"), and meta-header routing does not exist in this build
(providers route on the ORIGINAL Host). :8016 now proxies every request whole
to authentik-server:9000 preserving Host; the embedded outpost serves login
in place and forwards authenticated traffic back to :8017. Requires one proxy
provider per guarded host (provisioned live: search/hermes/prometheus/romm/
files + www, flows backfilled). Tests flipped from the meta-header claim to
the upheld contract: Host preserved, no absolute next= redirects.
2026-08-29 23:49:39 -07:00
Frank 8898740ca4 caddy: www holds the homepage; naked domain redirects at the TLS edge
The proxy provider matches external_host exactly, so Host=www matched zero
providers and Authentik answered its error page while the naked domain
served the homepage. Flip the topology: www.franksdank.cloud is the one
guarded vhost carrying the homepage through Anubis -> Authentik -> :8017;
franksdank.cloud (plus legacy names and the bare-IP catch-all) 308-redirects
onto www before any challenge. Tests flipped to assert the new contract.
2026-08-29 15:11:18 -07:00
Frank ef07e7d01a fix(caddy,authentik): auth guarded vhosts by proxying the whole request through authentik
The forward_auth target /outpost.goauthentik.io/auth/caddy does not exist
in authentik 2026.8 (verified live: 404 for every header combination; the
URL tree has no outpost auth prefix at all — the outpost worker dials back
to core over ws/outpost/<uuid>/ instead). Unauthenticated subrequests got
404, Caddy treated >=400 as 'not authenticated' and redirected browsers to
the dead path — users saw authentik's own Not-Found page.

Fix: guarded vhosts now proxy the whole request through authentik-server:9000
with X-authentik-meta-outpost/-provider/-app headers (the exact header set
this build's Traefik controller emits, adapted for Caddy). Unauthenticated
clients get the login flow served in place; once a session exists,
authentik proxies on to the provider's internal_host — now
http://caddy:8017, a new post-auth internal listener that routes straight to
the app. The chain terminates; nothing dials back into authentik.

Live DB changes applied alongside (postgres is the source of truth):
provider pk=1 authentication_flow=default-authentication-flow (was NULL —
the actual cause of the 404s) and internal_host=http://caddy:8017 (was
self-referential http://authentik-server:9000).
2026-08-28 22:28:32 -07:00
Frank 21fa38e6a8 anubis: fix image ref typo in deploy pull, health probe exec form, unit alias tag
Three compounding deploy-handler bugs found by live deployment:
- podman image exists/pull used a mixed-case ghcr.io ref that podman's
  reference parser rejects before any network I/O (repository name must be
  lowercase); all refs now lowercase — ghcr resolves case-insensitively.
- the unit runs the image under a localhost/ alias tag: this host's podman
  5.7 quadlet path fails container create with 'image not known' for
  registry-prefixed refs even when present in storage (verified live).
- the health probe was shell-form and the distroless ko-build image has no
  shell; HealthCmd is now a JSON exec-form array, which passes through the
  quadlet generator's argv assembly verbatim.

Tests updated to assert the alias-tag reality plus the pull guard.
2026-08-28 11:18:41 -07:00
Frank 91532a3805 anubis: repin the image ref to the lowercase ghcr path
podman's reference parser rejects mixed-case repository refs before any
network I/O ('repository name must be uppercase'); upstream's own install
docs pull lowercase (ghcr resolves case-insensitively). The first deploy
of this chain exit-2'd on that parse error — the image was not yet
pulled, so the guarded vhosts 502 until this deploy lands.
2026-08-28 07:05:24 -07:00
Frank 0ae3650414 anubis: bot-challenge proxy in front of the guarded vhosts (two-tier chain)
Bots and AI scrapers hit the SSO-fronted vhosts and eat Authentik auth
work for 403s. Anubis verifies clients first: DENY for known AI scraper
UAs, JS challenge for everyone else; verified traffic resumes the normal
Authentik forward_auth chain on Caddy's internal listener (:8016).

Chain: client -> Caddy (TLS) -> Anubis :8923 (verify) -> Caddy :8016
(auth + routing resumes; Host survives both hops) -> app. Reverse-proxy
through Anubis, not forward_auth: at the pinned tag a CHALLENGE answers
HTTP 200 + HTML, and forward_auth treats 200 as "authenticated" — the
page would pass through unchallenged. Exempt by construction (never
routed through Anubis): plex (plain HTTP legacy), auth.* (IdP login
must be reachable ungated), gitea (git smart-HTTP clients cannot solve
JS challenges), and every app with its own login.

Pinned ghcr.io/TecharoHQ/anubis:v1.13.0 — the tag was source-verified
before pinning: at this tag the policy parser is JSON-only with a
minimal bots[] schema (no imports, no CEL expressions), flags are the
config surface via flagenv (TARGET/POLICY_FNAME map to --target/
--policy-fname), and the image's exec-form HEALTHCHECK is inherited by
the quadlet unit (dodges the v1 generator's HealthCmd space-split quirk).

Rollback nuance: stopping anubis 502s everything behind the chain (the
public listeners proxy through it); a true rollback is reverting the
Caddyfile and redeploying, not just systemctl --user stop anubis.
2026-08-28 06:38:24 -07:00
Frank 914e251ebc authentik: boot-persistent quadlet conversion with deploy handler
The compose-era authentik trio died at every boot (rootless podman does not
auto-start bare containers). Convert the trio to Quadlet units installed by
deploy-remote.sh into ~/.config/containers/systemd/, enabled + WantedBy=
default.target:

- 3 unit files under dankclaw/authentik/quadlet/ (postgres -> server -> worker
  ordering via Wants=/After=; secrets stay in the git-ignored host .env,
  expanded by systemd at start time exactly like the gitea units)
- deploy_authentik() handler: install units, daemon-reload, retire stale
  compose-era containers only while units are inactive, enable once, then
  start in dependency order with podman-health gates between stages
- compose.yaml: authentik service definitions retired to a comment pointing at
  the quadlet dir (same pattern as searxng/open-webui/gitea/caddy)
- tests/authentik.bats rewritten from compose assertions to unit-file
  contract assertions (images, Exec roles, ordering, loopback-only publish,
  pg_isready probe, secret hygiene, dispatch routing)
2026-08-28 03:09:19 -07:00
Frank 5829a0bfe8 docs: rename docs/ -> serverAppConfigs-docs/ per system-wide PROJECTNAME-docs convention
All in-repo path references (AGENTS.md, README, old scripts, report artifacts,
test code paths) rewritten in the same commit; focused-test map extended for
.gitignore, old/scripts/*, dankclaw/comfyui/*, and report/* per this repo's
own convention ('add one before committing'). Obsidian wikilinks are
title-based and unaffected.
2026-08-28 02:20:06 -07:00
Frank e376d6a34e merge: Tiel-Coder-35B (dankserver) + Flash-Next 56B (dankclaw) + config-assertion test removal
Dankserver: ik_llama Ornith CRACK Q8_0 → stock llama.cpp Tiel-Coder-35B-A3B (qwen35moe hybrid MoE, --cpu-moe, 200700 ctx / 3 parallel)
Dankclaw: kingjones Q6_0 AGENT → Flash-Next 56B MoE (qwen4exp, PR #27742 merged into ROCmFPX fork, --fit off, 128k ctx)
Removed tests/frank_{small,large}_config.bats (config-assertion tests, not unit tests)
All 199 remaining tests pass. Both services live and verified.
2026-08-26 21:09:01 -07:00
Frank 215d74f6ce feat: swap both LLM services + remove config-assertion tests
Dankserver (frank-default-small):
- Stock llama.cpp (latest master, CUDA build) replaces ik_llama.cpp
- Model: peculiar-ragdoll/Tiel-Coder-35B-A3B UD-Q6_K_XL (30 GB)
  Architecture: qwen35moe, 40 layers (10 attn + 30 SSM), 256 experts/8 used
- CTX_SIZE=200700 (66900/slot × 3 parallel), Q8_0 KV (~2 GB cache buffer)
- --cpu-moe for expert offload, -ngl 999 (non-expert on GPU)
- mmap enabled; removed ik_llama-specific flags

Dankclaw (frank-default-large):
- ROCmFPX fork with PR #27742 (qwen4exp arch) merged and built
  at /home/dank/GitPackages/ROCmFPX/build-qwen38
- Model: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN (98.5 GiB)
  Architecture: qwen4exp, 48 layers (12 attn + 36 SSM), 512 experts/10 used
  QSA indexer, hyper-connections, PLE, Q4_0_ROCMFP4 quant
- CTX_SIZE=131072 (128k) × 1 parallel, f16 KV
- --fit off (auto-fit segfaults on qwen4exp hyper-connection config)
- Removed MTP draft, mmproj; kept Qwen-Sharp chat template + reasoning

Tests:
- Removed tests/frank_small_config.bats and tests/frank_large_config.bats
  (moved to .AGENTTRASH/). These were config-assertion tests that checked
  'is variable X set to Y' rather than testing the serve script's actual
  behavior. All 199 remaining tests pass.
- Updated test-changed.sh mappings and changed_tests.bats to reflect the
  removal.
2026-08-26 20:37:46 -07:00
Frank 868c8cfbca caddy: drop the systemctl enable; the quadlet generator already enables it
podman's quadlet generator creates the default.target.wants symlink from
the unit's [Install] section at every user-manager start (is-enabled
reports 'generated' for all healthy quadlets, pihole included). An
explicit 'systemctl --user enable caddy.service' fails with 'transient or
generated' and aborted the deploy after the unit was already installed.
2026-08-26 20:19:03 -07:00
Frank f3c1ac1823 pihole: also neutralize the /etc/sysctl.conf sysctl override
systemd-sysctl applies /etc/sysctl.conf AFTER all of sysctl.d/, so the
stale ip_unprivileged_port_start=80 line there beat every drop-in and kept
the runtime value at 80 after the first deploy pass. deploy_pihole now
comments out an active line in /etc/sysctl.conf (idempotent) and generalizes
the sysctl.d conflict scan to any file setting a value above 53.
2026-08-26 20:08:30 -07:00
Frank 8660209c35 caddy + pihole: make both boot-persistent and self-healing
Caddy left compose.yaml for a quadlet unit (caddy.service): joins
dankclaw_default for container-name upstreams, Pull=never for the local-only
image, Porkbun credentials via /home/dank/Containers/caddy/env written by
deploy from .env, bare-IP-catch-all health check. deploy_caddy installs,
enables, and starts the unit after retiring the bare container (state is in
host volumes).

Pi-hole: the repo now owns the sysctl drop-in
(dankclaw/pihole/sysctl/99-dnsmasq.conf, ip_unprivileged_port_start=53);
deploy_pihole installs it to /etc/sysctl.d/, stages conflicting later-sorted
drop-ins in the home trash, and applies 'sysctl --system' (the boot code
path). This fixes the 2026-08-26 crash loop (port-53 bind denied, restart
counter 1645) caused by a second drop-in resetting the value to 80.

Tests: new caddy_quadlet.bats seam; caddyfile/deployment_flow assertions
repointed from the removed compose service to the quadlet; pihole.bats pins
the sysctl drop-in and its deployment; test-changed.sh routes dankclaw/caddy/
and the pihole subtree to their seams. Docs: Caddy, Pi-hole, Dankclaw Stack,
work log.
2026-08-26 19:52:57 -07:00
Frank a6601e527e hermes-webui: bind 0.0.0.0 inside the container (host publish stays loopback)
Podman's port forwarder dials the container bridge IP; a container-internal
loopback bind made every tunneled connection reset.
2026-08-26 08:25:40 -07:00
Frank 105a4cdfef deploy.sh: accept hermes-webui in the top-level service inventory 2026-08-26 08:22:23 -07:00
Frank bb0404c310 hermes-webui: add Hermex server as loopback compose service on Dankclaw
- compose: hermes-webui runs bootstrap.py --foreground in the agent image,
  shares /home/dank/hermes-data (same uid mapping as the gateway), publishes
  127.0.0.1:8787 only; password from host .env
- planner/remote deployer: hermes-webui is an isolated Dankclaw service
- tests: deployment integrity seam (loopback, auth, volume, healthcheck,
  planner narrowing, deployer dispatch)
- docs: Hermes-WebUI page with the iOS SSH-tunnel access pattern
2026-08-26 08:17:27 -07:00
Frank 24c5eb4e28 frank-default-small: KV K default f16 (no hadamard), V q6_0 + hadamard
K cache back to default f16 full precision with K_CACHE_HADAMARD=0;
V cache at q6_0 with V_CACHE_HADAMARD=1. Env, serve script, bats, and
KB updated.
2026-08-26 05:53:01 -07:00
Frank 8f860d24a6 frank-default-small: 491520 context pool (163840 per slot x 3)
--ctx-size is the total pool divided among --parallel slots, so with
PARALLEL=3 the 160k target must be a 491520 pool (163840/slot) to
actually deliver 160k context per slot. Env, serve default, bats updated.
2026-08-26 05:48:28 -07:00