The vllm-radiance replica pair could not use both cards for one model: TP=2
needs a cross-card collective and the second R9700 cannot dispatch one. The
same wall blocks vLLM's pipeline parallelism, whose stage handoff is an RCCL
send/recv of the hidden state (distributed/parallel_state.py) -- a fact worth
writing down, because the obvious reading of the ask ('use PP to dodge the
atomics problem') is exactly backwards.
llama.cpp's layer split needs no collective at all: layers are distributed and
the boundary activation is a per-backend copy, once per forward pass. One
process, both cards, one model. The runtime is mainline llama.cpp at fb27a525
(the ref is pinned by DFlash2 AND qwen35 MTP; the ROCmFPX fork this host had
before has neither), built for gfx1201 against the private ROCm 7.14 toolchain
the earlier build already used, and bundled next to the binary.
Server: ajgazin Swift-Qwen3.8-27B-Uncensored UD-Q8_K_XL with the model's MTP
head in the file, plus the z-lab DFlash2 Q8_0 block-diffusion drafter at depth
7 (its block is 8, so 7 is the ceiling llama.cpp allows). Text only: vision
stays on frank-small.
Retired into the trash rather than deleted: the three vLLM quadlets, the
radiance entrypoint/healthcheck, the routing proxy and its tests, and the two
bench scripts that drove 'vllm bench serve' inside the retired image. The
deploy handler now stops them, waits for 7770 to be released, installs the
ROOT unit and refuses to start anything when the runtime or payload is
missing. The planner maps the scripts/ trio to frank-medium only, and the
scoping seam asserts the retired service dir is gone rather than merely
unmapped.
Dankclaw's endpoints are re-split. frank-large is the halogen-flash-server
quadlet again -- the 2026-09-13 unit file was never touched by the two-day
ik_llama.cpp interlude, so the runtime was still in the repo and its image is
still in local storage -- retuned to the shape asked for: HALOGEN_CTX 200192
with a 400,384-position pool and two slots, exactly two 200k conversations.
frank-small is new: mainline llama.cpp on the RTX 3090 serving
ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF at IQ3_S, using the -mtp build (it carries
blk.64.nextn.*, which is what lets --spec-type draft-mtp run on the main
model's own weights instead of a second 12 GiB copy) plus the BF16 vision
projector, as one 131,072-token slot with f16 KV on port 7771.
The host is shared now, and that is what the halogen retune pays for:
HALOGEN_MAX_TOK 24576 (arena 20.6 -> 16.4 GiB, ~4-5% prefill) and the vision
tower dropped (1.97 GiB) hand ~6 GiB back to the host, and the reserve rises to
20 to keep the fit check honest. Vision moved wholesale to frank-small.
Retired to scripts/.AGENTTRASH/: the ik trio (env, service, serve script), the
Qwen-Sharp template only that runtime read, and four benchmark harnesses bound
to retired runtimes. Because retiring a file makes any test that greps it pass
by absence, the seams assert the archive exists AND the live path does not.
Two things, found while checking whether the hermes trio still needed migrating.
1. **The phone-facing webui was exposed to the LAN.** Its unit runs with
Network=host, so there is no Podman port forwarder and the app's own bind is
the exposure. The compose service it replaced published
`127.0.0.1:8787:8787` (loopback); the unit bound `HERMES_WEBUI_HOST=0.0.0.0`,
and the LAN address answered 200 — password-gated, but on every interface,
which is not the documented "SSH tunnel is the sole ingress" posture. It now
binds 127.0.0.1, which serves the tunnel just as well (sshd dials
localhost:8787 from the host). The unit comment claiming "LAN-facing as
before" was wrong about "before", and is corrected.
2. **Dankclaw's compose.yaml is retired.** Auditing ownership showed every
running container on that host reports a PODMAN_SYSTEMD_UNIT label — the
hermes trio's units have owned their containers since 2026-09-12 — so the
three service blocks left in compose.yaml were dead weight that could only
create a duplicate container fighting a unit for a name. deploy_hermes was
already fully unit-based. The file is staged in .AGENTTRASH/; the planner
plans nothing for it, an unknown service on that target now errors instead of
silently no-oping through `podman-compose`, and DANKCLAW_COMPOSE_SERVICES is
empty.
The hermes-webui seam now reads the unit — it read compose, which is exactly how
the bind drifted while the test kept passing against a file nobody ran.
Dankserver's node-exporter has been dead since a reboot: `podman ps -a` showed
it `Exited (143)` and Prometheus' `node-dankserver` job has been scraping
nothing. It ran only from `dankserver/compose.yaml`, which is the failure mode
this repository's boot-persistence rule exists for (rootless Podman does not
restart bare containers after the stop that shutdown performs), and the compose
form bound the wildcard address — the unauthenticated-metrics-API-on-every-
interface posture already removed on Dankclaw.
- dankserver/observability/quadlet/node-exporter.container — mirrors the
Dankclaw unit, listening on exactly 127.0.0.1:9100 (host-local probes) and
192.168.0.235:9100 (Prometheus on Dankclaw), with Restart=always and
WantedBy=default.target.
- the compose service is deleted: one owner per container name.
- deploy_observability now installs from "$TARGET/observability/quadlet"; it
hardcoded the Dankclaw path, so a Dankserver unit could not be installed by
the shared dispatcher at all.
- planner: "node-exporter" joins the Dankserver service set, with a path rule
for dankserver/observability/**; the dispatcher's allowlist gains
dankserver:node-exporter.
Seams cover the silent halves: both listen addresses present and the wildcard
absent, the compose service gone, the allowlist entry present, and the planner
mapping the path to the service.
Grafana has been running with zero datasources and zero dashboards, so the
metrics scraped last commit were only reachable through the Prometheus API.
This wires the plumbing and one dashboard, both declared in
shared/grafana/provisioning/ and installed to /opt/grafana/provisioning by the
deployer (the unit already mounted that directory read-only, so nothing about
the container changes):
- datasources/prometheus.yml — Prometheus at http://prometheus:9090, the
container name, not the loopback publish, with a fixed uid so dashboards
reference it without a lookup.
- dashboards/llm-fleet.json — KV pool fill, queue depth, prefix-cache reuse
rate, e2e latency p95, and decode/prefill throughput, with an endpoint
variable covering both frank-medium and frank-large. Every series name was
checked against the live /metrics endpoint: this fleet already shipped one
comment naming a metric that does not exist, and a dashboard full of those
would be worse.
Dashboards stay editable in the UI (allowUiUpdates) because the panels are a
starting point, but the file is the source of truth and the deployer
overwrites Grafana's copy on every deploy.
Planner and seams follow the existing pattern: shared/grafana/** selects the
grafana service, and the tests pin the container-name address plus the
provisioning contract so a rename cannot silently empty the dashboard.
Rootless Quadlet on dankserver serving
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (the radiance stack's
designated alternate target; ships its fp8 MTP head, no rewrite step) with
the DFlash2-FP8 drafter at SPEC=7 on the 2x R9700s (TP=2, host network,
:7770, alias frank-medium unchanged).
- dankserver/frank-medium/: quadlet unit, container entrypoint (fork of
serve-mxfp4.sh's patch block at pinned clone commit 97353293), healthcheck
- dispatcher retires the root llama.cpp unit, re-tags the registry image to
the localhost alias, installs the quadlet, waits on /health
- planner maps dankserver/frank-medium/* to frank-medium; scripts-era
mappings removed
- ROCmFPX-era env/serve/unit/download scripts staged in scripts/.AGENTTRASH
with .rocmfpx suffix; seams updated (frank_medium_quadlet.bats new)
The RTX 3090 left Dankserver; two AMD Radeon AI PRO R9700 cards (Navi 48,
gfx1201, 32,624 MiB each) replaced it. FreeToken/CUDA cannot run there at all,
so the `frank-small` service is retired and `frank-medium` takes the endpoint.
New service: charlie12345/ROCmFPX llama.cpp (built for gfx1201 against the
fork's self-contained ROCm 7.14 toolchain, so the host's stale /opt/rocm-6.3.3
never participates) serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF --
a Q8_0_ROCMFPX_AGENT (ftype 115) checkpoint that stock llama.cpp cannot load.
Placeholder config: --split-mode tensor across both cards, 131072 context,
1 slot, f16 KV, MTP draft head on ROCm0.
deploy-remote.sh grows retire_root_llm_small(): the CUDA-era unit is stopped,
disabled and moved to ~/.AGENTTRASH/20260915-frank-small-retire/host-units
before frank-medium starts, because `systemctl disable` has been observed to
delete the unit file itself on this host (2026-08-26 incident).
The retired frank-small repo files (env, unit, serve script) move to
scripts/.AGENTTRASH/, and the planner, dispatcher, service allow-list and the
deployment test seams now name frank-medium.
prometheus, grafana, loki, alloy and node-exporter ran only from
dankclaw/compose.yaml. Rootless podman does not restart bare containers, so
they did not survive the 2026-09-11 reboot for the amd_iommu=off change and
had to be restored by hand — the same gap that had already cost the fleet
Caddy and Pi-hole, which is why the boot-persistence rule exists.
Each is now a rootless Quadlet unit under dankclaw/observability/quadlet/,
installed and dispatched by deploy_observability in deploy-remote.sh, with
the compose definitions retired so only one owner exists per container name.
Images, mounts, arguments and container names are unchanged, so Caddy's
proxies and Prometheus' own scrape config keep resolving.
Two posture changes, both about the unauthenticated surface:
* The three HTTP UIs published on 0.0.0.0 with no authentication of any kind
(Prometheus even with the lifecycle endpoint enabled). They now publish on
127.0.0.1: the browser path is the guarded franksdank.cloud vhost, Caddy
reaches them by name over dankclaw_default, and the host-side homepage
status timer probes loopback. Nothing legitimately used the LAN port.
* node-exporter bound :9100 on every interface; it now binds exactly its two
consumers, loopback (homepage timer) and the LAN address (Prometheus, which
runs in a container and cannot reach the host's loopback).
alloy gains a reachable health endpoint: its UI listens on the container's
own loopback and alloy 1.x refuses a config block for it, so the compose
service had nothing to probe — the homepage status card for alloy read 'down'
continuously. The unit moves the UI with a command-line flag and publishes
127.0.0.1:12345.
grafana keeps its Authentik OAuth client secret without receiving the whole
.env: systemd expands the value from a [Service] EnvironmentFile, which never
reaches the container.
frank-large moves from the kingjones30/ROCmFPX llama.cpp build to
peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b
.hgn 115.55 GiB + quality overlay). Upstream measures this engine at
roughly 4x the prefill throughput of the llama.cpp family on the same
gfx1151 silicon, and prefill dominates any prompt with real context.
Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/),
engine on container loopback :8730 (never published), OpenAI front-end
:8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client
renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB,
conservative for a shared 121 GiB host), 4 slots, host reserve 24,
vision tower on, weights pinned, all other flags at vendor defaults.
Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable
/home/dank/models/halogen.
The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/
rather than deleted; the STRIX payload and the ROCmFPX build stay on
disk as the rollback target. deploy_llm_large becomes a user-quadlet
handler: it retires the old ROOT unit (moving its checkout symlink into
the home trash, never rm) and gates the first start on the download
rather than reporting a failure for a fetch that is working exactly as
designed.
frank-npu is retired in the same change. The kernel cmdline moves from
iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured,
and that takes the XDNA2 NPU with it: without PASID/SVA the driver
cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are
repointed (Hermes vision -> frank-large, Hermes title generation ->
frank-small, harness vision -> frank-large, tiny -> frank-small, task ->
frank-large:medium) and its provider blocks are removed from both the
harness config and Hermes's live config. Audio input leaves the fleet
with no replacement; image input survives on halogen's vision tower.
Docs: new halogen-flash-server runbook, new IOMMU decision record, and
frank-npu rewritten as a retirement record. AGENTS.md topology, runtime
sections, benchmarking policy (frank-large is a user unit now) and the
harness model-config section are corrected.
FrankBook-Pro becomes the only *arr/qBittorrent deployment. Every fleet
container (qB + five arrs) now shares gluetun's netns, so all egress —
torrent, indexer APIs, metadata lookups — exits the WireGuard tunnel by
construction (AllowedIPs=0.0.0.0/0); the arrs' own ports publish on the
gluetun unit, none on the arr units themselves.
- dankserver/compose.yaml: remove the six arr services (the Aug 20 compose
copy — never healthy, superseded; config dirs trashed on the host)
- dankclaw/Caddyfile + shared/homepage/services.json: arr vhosts and
probes repoint 192.168.0.235 -> 192.168.0.112 (FrankBook-Pro)
- frankbook-pro units: five arrs join gluetun netns (Wants/After/Network);
gluetun publishes 9696/8989/7878/8686/8787 alongside 8080/6881
- tests/frankbook_quadlet.bats: new VPN egress seam — every arr+qB unit
must share gluetun's netns and publish no port; gluetun must publish the
whole fleet's ports
- deploy-plan.py: gluetun changes carry the whole netns family (a gluetun
recreation kills all six joiners); dankserver compose set loses the arrs
- AGENTS.md: deployment is complete only when unit tests AND the Playwright
E2E suite both pass
frank-default-large -> frank-large (Dankclaw :7770), frank-default-small ->
frank-small (Dankserver :7770), frank-default-sightsound -> frank-npu
(Dankclaw :7771). Units, env files, serve scripts, deploy-plan mappings,
deploy-remote dispatch (stop/disable old units + relink + enable new), and the
test seams move together. Retired frank-default-npu.* artifacts moved to
scripts/.AGENTTRASH/ (tracked-in-history; ignored on disk).
Served aliases become frank-large / frank-small via --alias; the NPU endpoint
serves the whole FLM registry, so its rename changes no served tags.
- deploy_plex(): install unit, stop+mask native plexmediaserver.service on
first cutover (mask, never disable), gate on /identity for 180s
- deploy_tautulli(): install unit, stop+disable snap on first cutover,
seed config.ini with PMS_Web_Host=plex only if absent
- allow-lists (deploy.sh valid_service, deploy-remote.sh service_is_allowed),
planner routing (SERVICE_ORDER, DANKSERVER_SERVICES, quadlet_services),
test-changed cutover case, quadlet seam enumeration + new
userns/PUID=997/PGID=984 ownership-contract test
Seven units installed INSIDE the Mac VM (/home/core/.config/containers/
systemd) by a new deploy_frankbook_stack handler: gluetun (ProtonVPN
WireGuard, env-file credentials, publishes qbit's 8080/6881), qbittorrent
(Network=gluetun.container, verbatim /Volumes data roots, first-boot
config seed with fresh machine-local WebUI password and the Finished
Torrents category), and prowlarr/sonarr/radarr/lidarr/readarr (grab-only,
config migrated from dankserver at provisioning, readarr pinned to
amd64-1.0.980). deploy.sh routes the new target through the ssh-config
alias with REMOTE_SHELL=/usr/local/bin/bash (macOS bash 3.2 lacks
associative arrays); the planner learns FRANKBOOK_SERVICES and the
gluetun->qbittorrent netns pairing; new seam test pins the contract.
Cutover of the six surviving dankserver services from compose-era bare
containers to boot-persistent Quadlet units (WantedBy=default.target; podman
5.x v1 generator honors it at user-manager start).
Units are faithful to live inspect ground truth (image pins = running tags,
not compose inventory):
- romm trio shares one data root: /home/dank/Containers/romm mounted whole at
/romm; mariadb datadir and redis data dir live inside it. Aliases mariadb/
redis preserved on dankserver_appnet via Network=...:alias=. Secrets stay in
the host-local env file (EnvironmentFile + ${VAR} expansion in the generated
ExecStart, gitea precedent). copyparty passes its admin account via
Exec=-a ${COPAY_ACCOUNT}:${COPAY_PASSWORD} (Exec= not Command=: the v1
generator hard-fails on Command=); entropy stays rw only there.
- jellyfin keeps every creation-time env as Environment= lines (zero drift);
Device=/dev/dri + GroupAdd=keep-groups preserve hardware transcoding.
- calibre-web pins the exact running image tag (id 986277563e76 =
0.6.27-ls397), pasta network preserved.
Deployer: first-cutover handlers retire the old container, rescue unshadowed
anonymous-volume content (podman volume export + tar --skip-old-files), mv
data roots on the same fs, then start units in dependency order with health
gates. Re-deploys restart only the selected unit. deploy-plan.py maps compose
keys mariadb/redis to the new unit names; allowlists updated. New seam test
tests/dankserver_quadlet.bats pins boot-persistence + v1-generator contract;
test-changed.sh maps the new dirs to it. Full suite green (209 tests).
New Dankclaw service frank-default-sightsound: FLM 1.0.1 serving the
Gemma4-E4B-IT-NPU2 pack (vision+audio+reasoning, 131072 ctx) on the XDNA2
NPU at port 52626. Model choice rationale: gemma4-it is the only NPU2 model
family with BOTH image and audio towers; the uncensored candidate qwythos:9b
is text-only, so sight+sound wins over uncensored here.
Deployer wiring: planner maps env/unit/serve files to the new service name;
deploy-remote gains deploy_llm_sightsound (relink + daemon-reload + enable +
restart + health gate on /v1/version — FLM has no /health route). Unlike the
retired frank-default-npu relink-only path, this unit is boot-enabled.
The quadlet-dir allowlist silently dropped every change under
dankclaw/authentik/ — the new outpost unit (and its siblings) now route
by exact unit name, mirroring the searxng/gitea pattern. Adds
authentik-outpost to the service inventories; it is a quadlet unit, not
a compose service, so it stays out of the compose set.
Bots and AI scrapers hit the SSO-fronted vhosts and eat Authentik auth
work for 403s. Anubis verifies clients first: DENY for known AI scraper
UAs, JS challenge for everyone else; verified traffic resumes the normal
Authentik forward_auth chain on Caddy's internal listener (:8016).
Chain: client -> Caddy (TLS) -> Anubis :8923 (verify) -> Caddy :8016
(auth + routing resumes; Host survives both hops) -> app. Reverse-proxy
through Anubis, not forward_auth: at the pinned tag a CHALLENGE answers
HTTP 200 + HTML, and forward_auth treats 200 as "authenticated" — the
page would pass through unchallenged. Exempt by construction (never
routed through Anubis): plex (plain HTTP legacy), auth.* (IdP login
must be reachable ungated), gitea (git smart-HTTP clients cannot solve
JS challenges), and every app with its own login.
Pinned ghcr.io/TecharoHQ/anubis:v1.13.0 — the tag was source-verified
before pinning: at this tag the policy parser is JSON-only with a
minimal bots[] schema (no imports, no CEL expressions), flags are the
config surface via flagenv (TARGET/POLICY_FNAME map to --target/
--policy-fname), and the image's exec-form HEALTHCHECK is inherited by
the quadlet unit (dodges the v1 generator's HealthCmd space-split quirk).
Rollback nuance: stopping anubis 502s everything behind the chain (the
public listeners proxy through it); a true rollback is reverting the
Caddyfile and redeploying, not just systemctl --user stop anubis.
Frank