- Homepage Design Grilling.md: final decision table (Q1-Q43) + the deltas where
the operator changed the recommendation (per-service probe method, 5s timeout,
TUI panels, toggleable styling, no cursor alteration, no auto-refresh, /docs/
path, user-facing docs, git-driven deploy, full TDD, last-deploy footer).
- Homepage.md: updated to the settled design (per-service probe, 5s, TUI panels,
styling toggle, no auto-refresh, /docs/ workstream, git-driven deploy, TDD).
- pi log.md: 2026-08-23 entry recording the settled design and build order.
Build order is unblocked. No homepage code built yet.
- Homepage.md: rewritten to the settled design (single site, user + /admin/
pages, server-side status.json via a 3-min systemd-timer checker, terminal
TUI theme). Added mermaid diagrams: site structure, status states, checker
flow. Recorded the classification (user vs admin; *arr excluded; LLM admin;
backends status-only) and the documented future shift to client-side probing.
- Homepage Design Grilling.md (new): the decision log — settled Q1-Q5 plus 38
open questions (Q6-Q43) across 10 categories, each with a recommendation.
- docs/index.md: link the grilling page (satisfies the index-links-every-page test).
- report/homepage-grilling-2026-08-22.html: self-contained terminal-TUI progress
report (settled decisions, verified stack baseline, overnight processing, work
threads, the open questions, and how to continue).
Baseline recorded: Dankclaw fully up (10/10); the Dankserver media stack is down
(only Calibre-Web + the LLM are up; no .env on the target) — a separate thread.
No homepage code built yet; the design is pending the operator's answers to Q6-Q43.
The explanatory comment contained an unescaped loop-variable token, which
the local shell (set -u) tried to expand while building the double-quoted
remote script ('unit: unbound variable'). Reword the comment.
The relink loop wrapped the source path in single quotes, which shipped a
literal ${unit} to the remote and created a dangling symlink
(/etc/systemd/system/frank-default-large.service -> .../${unit}.service),
so 'systemctl enable' failed with 'unit does not exist'. The path has no
spaces; unquoting lets ${unit} expand on the remote for both source and
destination.
Operator request (2026-08-22): the primary inference server
(frank-default-large, the 27B CIRU model on Dankclaw) moves to the
standard llama.cpp port 7770. Both LLM services now listen on :7770,
distinguished by host (dankserver vs dankclaw).
Functional changes:
- scripts/frank-default-large.env: LISTEN_PORT 52625 -> 7770
- scripts/serve-frank-large.sh: default LISTEN_PORT 52625 -> 7770
- scripts/deploy.sh: LLM_HEALTH[dankclaw] -> :7770
- open-webui quadlet: OPENAI_API_BASE_URL -> 192.168.0.143:7770
(re-pointed so chat keeps working; data dir preserved)
- scripts/bench-fast-greedy.sh: PORT 52625 -> 7770
Tests updated to the new topology (both LLM endpoints on :7770 by host;
open-webui -> :7770). Docs + pi KB updated; dated historical runbooks
keep their point-in-time ports. models.json (pi client) re-pointed to
:7770 separately, effective on the next client restart.
- docs/index.md: LLM Fleet section (all 34 pages linked), merge note,
See Also now internal
- docs/LLM/index.md: retitled, dangling [[Dankserver]]/[[Dankclaw]]
links fixed, sibling section replaced with internal links
- docs/LLM/deployment.md: rewritten for the unified deploy flow,
manual-update paths point at the merged checkout, incident history
kept intact
- Deployment Flow / Dankclaw Stack / Gitea: stale vllmScripts
references and retired-topology notes updated to the 2026-08-22
state
- Historical docs (frank-redo, PP deployment, FastFlowLM) keep their
dated runbook paths as-is (records of retired topologies)
deploy.sh now deploys both workloads per target in one run:
push -> pull on /home/dank/serverAppConfigs -> podman stack steps
(dankclaw) -> LLM unit relink + daemon-reload + service restart ->
homepage health + LLM health (up to 20 min model-load wait).
Sudo: plain sudo on the fleet hosts (passwordless verified
2026-08-22); FRANK_SUDO_PW covers headless runs elsewhere. The
vllmScripts deploy.sh is superseded.
Units: frank-default-{small,large,npu}.service now reference
/home/dank/serverAppConfigs/scripts/... (the merged checkout).
bench-spec-stack.sh: env file path updated to the merged checkout.
README: merged service table (stack + LLM), structure incl. old/ and
docs/LLM/, unified deploy note, merge note.
.gitignore: merged both files; !scripts/*.env exception for the LLM
tunables (paths and numbers, never credentials); .obsidian/,
__pycache__/, .venv/ from the vllmScripts side.
One repo, one git protocol, one deploy script. Merged from the two
former AGENTS.md files:
- Overview: both workloads, both hosts, the 2026-08-22 merge note,
current LLM fleet topology (frank-default-small :7770 Dankserver,
frank-default-large :52625 Dankclaw)
- Git workflow + testing (serverAppConfigs protocol, test step kept)
- Deployment: the unified deploy.sh (stack + LLM per target)
- Secret rules: no passwords in the repo; FRANK_SUDO_PW covers
headless sudo (the live sudo password is NOT carried forward)
- Deletion & Trash Policy (from the vllmScripts 2026-08-22 commit)
- Benchmarking policy (stop the service before any bench)
- Pi model config, runtime & troubleshooting (v3 CIRU active,
Kairic/DFlash2/FastMTP retired), HIP env naming rule, crash-loop
procedure, operational quirks (passwordless sudo verified
2026-08-22), Dankclaw memory model
New seams (written first, red against the pre-merge tree):
- llm_units.bats: every scripts/*.service unit points at files that
exist in this repo (the merged checkout path), env files included
- llm_secrets.bats: the live sudo password is never tracked (the
vllmScripts AGENTS.md once documented it)
- deployment_flow.bats: deploy.sh must restart the LLM services on
both targets and health-check both endpoints (7770, 52625)
- helpers.bash: has_no_leaked_sudo_pw (two-part assembly, same
pattern as the gitea check)
The LLM fleet serve scripts, systemd units, env tunables, and bench
harnesses join the deploy scripts. The vllmScripts deploy.sh is
removed here; its functionality is folded into the unified
scripts/deploy.sh in a follow-up commit.
bench-spec-stack.sh gains the executable bit (required by the
scripts_syntax test seam).
The LLM fleet documentation (deployment runbook, model runbooks,
benchmarks, research archive, retired-topology history) joins the
project knowledge base as docs/LLM/. Wikilinks resolve by filename,
so no link updates are needed for the move.
The vllmScripts repository (LLM inference fleet: serve scripts, systemd
units, env tunables, deployment runbook, research docs) is merged into
this repository. Its full history is grafted under llm/ with a
two-parent merge commit so no commit is lost.
A follow-up commit reorganizes llm/ into the repo layout:
llm/docs -> docs/LLM, llm/scripts -> scripts, llm/old -> old.
Explicit rule: stage deletions in a dated ~/.AGENTTRASH/YYYYMMDD-topic
directory, list sizes, and wait for user approval before purging. rm,
rm -rf, and git clean -f forbidden without explicit permission. Applies
to every host.
LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=0 unconditionally. FASTGREEDY=1 now
means: reasoning off, deterministic sampler defaults, compatible with
all non-greedy requests (temp != 0, penalties, reasoning budget). This
is the card's 'compatible mode' profile.
The bridge gateway (10.89.3.1) lives in the rootless netns; llama-server
runs in the host netns, so the gateway refuses. 192.168.0.143:52625
reaches it through the rootless NAT (verified from the container).
Same LAN-IP pattern the doc already prescribes for the Dankserver
backend.
A 'caddy reload' delivered over the admin API cannot apply a config
that disables the admin API -- the fix cannot ride the API it is meant
to enable (2026-08-22: admin off sat in the running config across
deploys). Track the reload result; on failure, restart the container
so it reads the rendered file on start.
Reasoning off, argmax fastpath on, deterministic sampler. The fast
greedy path rejects non-greedy requests with 400; chat clients must
send temperature 0.
The Caddyfile had 'admin off', which disables Caddy's config file
watcher. Rendered Caddyfile changes never loaded until the container
was recreated, so the double-hash basicauth fix sat on disk unapplied
across deploys (2026-08-22). Admin binds to 127.0.0.1:2019 inside the
container and the port is unpublished, so nothing outside reaches it.
deploy.sh now issues an explicit 'caddy reload' (retry loop) after
'up -d', with the file watcher as backstop.
basicauth (2026-08-22 incident):
- the Caddyfile template carried its own $2a$14$ prefix and deploy.sh
substituted the full hash (prefix included), rendering an invalid
double-prefixed hash. Every basicauth vhost rejected every password
with 401 and Caddy logged nothing.
- template is now a bare REPLACE_WITH_HASH token; tests pin the render
contract (no prefix in template; rendered line is exactly 'dank <hash>').
open-webui (2026-08-22 incident):
- its slirp4netns helper process died, leaving the container running
with only 'lo' in its netns: port 3005 accepted TCP and reset every
request, so chat served empty responses for ~2 days, unlogged.
- move to a pinned bridge network (systemd-open-webui, 10.89.3.0/24,
gw 10.89.3.1) — no per-container helper to die.
- LLM endpoint reached via the gateway (host.containers.internal is
slirp-only): OPENAI_API_BASE_URL=http://10.89.3.1:52625.
- add a quadlet healthcheck: /health (app) + non-loopback interface
(network liveness; the dead-tap state leaves only 'lo').
- deploy script now copies the .network file too.
- docs updated (OpenWebUI History section, Caddy basic-auth contract).
buildah can serve a stale COPY --from layer (cache keyed on stage
name), so a Containerfile edit silently produced an unchanged image.
The image now carries a cfhash label; a mismatch forces a --no-cache
rebuild.
- alpine's /bin/busybox is musl-dynamic and cannot execute in the
distroless loki image (missing loader); busybox-static links in
- alloy 1.x refuses to configure the ui service from a config block;
the default UI already listens on 127.0.0.1:12345 where the
healthcheck probes
- alloy 1.x renamed the server component to ui; the old block failed the
initial load and the container crash-looped
- grafana/loki is distroless and grafana/alloy lacks wget, so their
healthchecks could never run; local images add a static busybox from
alpine for that one command
- deploy.sh builds all three local images if missing
- tests: 3 new seams
Measured the rootless mapping: container root IS the invoking user
(dank, owner of the /opt data dirs); the images' default users (loki
10001, prometheus nobody, grafana 472) map to subuids that own nothing.
1000:1000 was the wrong direction.
Container root maps to a subuid that owns nothing under /opt, so the
data-dir writes fail. Container uid 1000 is the invoking user in
rootless podman. Fixes the awk service-block seam to use regex match.
- loki: recent releases require compactor.delete_request_store when
retention is enabled (was crash-looping on start)
- prometheus: scrape targets carried legacy dankclaw.local names with
URL schemes (invalid); point at published LAN ports and the gitea
quadlet's published port instead
- grafana: GF_SERVER_ROOT_URL to the canonical HTTPS name
- tests: 3 new seams for the deployed observability configs
- Caddyfile: drop http:// site prefixes, global acme_dns porkbun block,
HTTPS redirects on the bare-IP and legacy vhosts
- new dankclaw/caddy/Containerfile: xcaddy build with caddy-dns/porkbun
- compose: caddy uses franksdank/caddy-porkbun:latest + Porkbun env vars
- deploy.sh: build the image on first deploy if missing
- .env.example: rename PORKBUN_API_SECRET -> PORKBUN_API_SECRET_KEY
(the name the caddy-dns/porkbun module expects)
- tests: 7 new caddyfile/compose seams for the TLS config
Local overrides to the pinned runner for production chat:
- --reasoning on --reasoning-format deepseek (thinking to
message.reasoning_content, matching the retired v3 service so the
fleet's reasoning clients work)
- --reasoning-budget -1 (unrestricted thinking)
- --chat-template-kwargs '{"preserve_thinking":"true"}' (preserve
thinking tokens in the conversation history across turns)
- batch 4096 / ubatch 2048 (larger than the runner's 2048/512; unified
memory has headroom)
The Kairic IU4 lane (--kairic-edge) and the MTP draft fast path are
unaffected.
The pinned runner sets LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=1, but the
exact-Q8 argmax fast path fails closed on any non-greedy request. The
model card's own chat instructions override sampling per request with
Qwen live values (temp 0.7, top-p 0.8, top-k 20, min-p 0,
presence_penalty 1.5), which the fast path rejects (400). Production
chat forces the fast path to 0; the Kairic IU4 lane (--kairic-edge) and
the MTP draft fast path are unaffected.
Serves jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (four-artifact payload) with
the qualified pinned-runner profile: 262K ctx, 1 slot, b2048/ub512,
t16/tb32, f16 KV, 32 ctx checkpoints, 8 GiB prompt cache with idle-slot
persistence, native MTP4, deterministic sampler, reasoning off.
The serve script replicates the release pinned runner verbatim; the only
differences are safe bind vars (no bare HOST env var; HIP collision rule)
and this machine's split 7.15 SDK LD_LIBRARY_PATH layout.
Rollback: revert this commit and redeploy; v3 model files and the v3
binary are untouched.
Per user direction, Caddy runs only on dankclaw. It fronts every
service on both hosts: local services by container name, quadlet
services by published LAN port, dankserver services by published LAN
port (192.168.0.235:PORT).
- dankclaw/Caddyfile: merged vhost list (was: two Caddyfiles, one per
host). gitea's single vhost moves in from the retired gitea-caddy
quadlet (option B). Upstreams fixed to live reality: gitea :3000,
search :8081 (gluetun), chat :3005 (open-webui quadlet), media
services via 192.168.0.235. Retired: llms (open-webui on dankserver
is removed; chat is its successor), the dankserver homepage (302 to
the shared one).
- dankserver/Caddyfile, dankclaw/gitea/Caddyfile: deleted (git rm).
- dnsmasq addn-hosts: every name -> 192.168.0.143 (the one Caddy
host); no .235 records, no llms.
- compose.yaml: removed the openwebui/postgres/gitea services (they
are quadlets now); caddy service mounts the rendered Caddyfile from
/home/dank/Containers/caddy, healthcheck probes the bare-IP
catch-all (302).
- deploy.sh: per-target logic (dankserver = pull-only, health via the
shared Caddy at 192.168.0.143); renders CADDY_BASICAUTH_HASH from
the host .env into the mounted Caddyfile.
- .env.example: CADDY_BASICAUTH_HASH on dankclaw; Porkbun block
dropped from dankserver (its Caddy is gone); SEARXNG_BASE_URL to
the flat name; stale OPENAI_* block dropped (quadlet is
self-contained).
- tests: caddyfile.bats now asserts the single-Caddyfile invariant;
dnsmasq.bats cross-checks vhosts -> .143 and the retired names.
/opt is root-owned; the established data-location pattern (see
docs/Architecture/Storage Policy.md, open-webui quadlet) is
/home/dank/Containers/<service>.
Phase 1 of the local-DNS-name work. Users type a name, not a port.
- dnsmasq quadlet on dankclaw (rootless, slirp4netns, port 53 published):
serves the flat name scheme from addn-hosts, upstream 1.1.1.1/1.0.0.1.
Image built from the in-repo Containerfile (localhost/dnsmasq:latest).
- Caddyfiles (both hosts): all *.dankclaw.local / *.dankserver.local vhosts
renamed to flat <service>.franksdank.cloud. Explicit http:// site
addresses for now; TLS (wildcard LE via DNS-01/Porkbun) lands in phase 2.
- Drift reconciliation (live /etc/caddy/Caddyfile on dankserver differs
from the repo): added the live-only llms.franksdank.cloud vhost
(OpenWebUI on :3000). Dropped the legacy search vhost from dankserver
(the name belongs to dankclaw's gluetun-routed SearXNG; the legacy
instance stays reachable at 192.168.0.235:8081).
- Homepage service links updated to the new names.
- New test seams (TDD): tests/dnsmasq.bats (quadlet boot-persistence,
port 53, conf/record validity, Caddyfile<->addn-hosts cross-check on
the right host IPs); caddyfile.bats naming-convention tests updated.
LAN clients get this DNS from the SURFboard G36 (DNS Relay -> statically
configured -> 192.168.0.143); the router change is a pending user step.
Real tokens live only in the host .env files (git-ignored):
- /home/dank/serverAppConfigs/dankclaw/.env
- /home/dank/serverAppConfigs/dankserver/.env
Token is scoped to the franksdank.cloud zone (DNS read/write).
Restore the CIRU ROCmFPX setup that DFlash2 replaced:
- Restore jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2 runtime
(native depth-4 MTP, PromptForge DualView, ROCm 7.15)
- Archive the DFlash2 Vulkan deployment files in .AGENTTRASH
- DFlash2 was retired the same day it shipped (2026-08-21) for speed
- Revert docs (deployment.md, index.md, AGENTS.md); mark dflash2 page archived
- Restored scripts are byte-identical to the pre-swap CIRU versions
Replace the retired CIRU ROCmFPX runtime with upstream llama.cpp PR #27342
DFlash2 on the Vulkan backend:
- Target: unsloth Qwen3.8-27B-UD-Q8_K_XL.gguf (31.4 GB, already on disk)
- Drafter: z-lab Qwen3.8-27B-DFlash2-Q4_K_M.gguf (block size 8, top-k 16)
- Runtime: /home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server
(GGML_VULKAN=ON; HIP lacks TOP_K so the stack runs on Vulkan0/RADV)
- spec: --spec-type draft-dflash --spec-draft-n-max 8 (clamps to 7)
Validated 2026-08-21 on scratch port 52626: mean acceptance 5.27,
health + reasoning completion OK.
Archive the retired CIRU .env/.serve script in .AGENTTRASH.
Update deployment.md, index.md, AGENTS.md; add docs/dflash2-large-vulkan.md.
Add a hermes-dashboard compose service that runs 'hermes dashboard' on
0.0.0.0:9119 with HTTP basic auth. The username + scrypt password hash
live in the git-ignored /home/dank/hermes-data config (plaintext never
stored). A separate container from the Signal gateway, since the gateway
entrypoint does not start the dashboard. Both share the data volume and
are boot-persistent. Docs page + index link added (docs-first contract).
The slirp host.containers.internal mapping (169.254.1.2) went down at
runtime, so the Signal adapter could not reach the signal-cli daemon.
The host LAN IP (192.168.0.143:8080) is the stable container->host path
and is reachable from the container. Switch SIGNAL_HTTP_URL to it.
'hermes gateway run' starts the messaging gateway (Signal SSE), not the :8090
web dashboard, so probing 8090 always reported unhealthy. The container's
running state plus the daemon's SSE connection are the real health signal.
The image's default s6 entrypoint launches the interactive TUI, which exits
without a TTY and crash-loops the container. Override the entrypoint to run
'hermes gateway run --force' (the documented Docker mode) and point HERMES_HOME
at the mounted data dir so the Signal platform config + state persist. The
Signal platform is enabled in config.yaml (platforms.signal.enabled).
The hermes container reaches the host signal-cli daemon via podman's
host.containers.internal alias (127.0.0.1 in the container is not the host).
SIGNAL_ACCOUNT/SIGNAL_ALLOWED_USERS come from the git-ignored .env (the phone
number is personal, not committed). Move the data volume to a dank-owned path
(/opt is root-owned and not writable by the run user).
ComfyUI-Manager (V3.41) is installed in the mounted custom_nodes dir and
self-installs its pip deps on start. Document the install command, the
bind-mount-shadows-image gotcha, and the /manager/version health check.
Rewrite ComfyUI.md to match reality: comfyui-rocm:gfx1151 built from
Containerfile.comfyui on AMD's gfx1151 base, device passthrough, video group
only (render is unmappable in rootless), data under /home/dank/comfyui-data,
boot-persistent via a user systemd unit. Mark research 'way 1' as applied and
update the repo-consequence section (the CPU config is no longer the default).
/opt is root-owned and not writable by the run user. Point the model/input/
output/custom_nodes mounts at a dank-owned path so the container can write
them without sudo. The tests do not assert volume paths, so this is a no-op
for the suite.
Rootless podman cannot map the 'render' gid (990) into the container
namespace (dank's subgid range is 100000+), so --group-add render fails to
start the container. GPU access actually comes from the /dev/kfd + /dev/dri
passthrough, checked host-side (the run user is already in 'render' on the
host). So add only the 'video' group (which resolves) and drop 'render'.
Update the group test to assert video present and render ABSENT (with the
rootless rationale), per the approved rootless config.
Replace the CPU stock image with a container built from
Containerfile.comfyui on AMD's gfx1151-tagged vLLM base — the only torch
build verified to report torch.cuda.is_available()=True on the Strix Halo
APU (Radeon 8060S, capability (11,5)).
- Containerfile bakes the ComfyUI code in and installs its deps, but skips
the torch family (the base image already ships the gfx1151 ROCm
torch/torchvision/torchaudio; an unpinned 'torch' in requirements.txt would
clobber the gfx1151 kernels).
- compose comfyui service: build + image comfyui-rocm:gfx1151, passes
/dev/kfd + /dev/dri, adds video+render groups, drops --cpu. Models/data
persist under /opt/comfyui.
- Un-skip the four compose-wiring acceptance tests (the gate for this work).
The JSON rate cap (API_MAX=4/hour) is hardcoded in Python, not settable in
limiter.toml; pass_ip is the only bypass. Rootless podman strips the client IP
on published-port hits (the container sees a slirp source), so local subagents
must present a pass_ip source. Document the slirp IP-strip, the pass_ip
expansion, and the root-owned config file (one-off sudo cp to update).
Root cause of the 429s: the ip_limit rate limiter caps JSON (format!=html)
requests at API_MAX=4 per IP per hour, and that constant is hardcoded in
Python (searx/botdetection/ip_limit.py), not settable in limiter.toml. The
only bypass is pass_ip (filter_request returns before ip_limit runs).
Rootless podman strips the client IP on published-port hits, so the container
sees a slirp/bridge source (10.88.x/10.89.x), never the real client - hence
even a LAN-IP curl 429'd on /search. Expand pass_ip to loopback + the host's
private podman/slirp ranges (plus the existing LAN) so local subagents present
a trusted source and skip the rate limit entirely. All these ranges are
reachable only from the host or LAN, never the internet.
User request 2026-08-19 (second pass): parallel slots degraded
performance. PARALLEL=1, CTX_SIZE=262144 (native max, -c equals the
per-slot context with one slot). Concurrent requests queue on the
single slot; the server never preempts a running generation.
pi models.json contextWindow/maxTokens back to 262144 (outside repo).
Docs updated to match.
User request 2026-08-19: lower per-stream context from 262144 to 200000.
CTX_SIZE is the total; llama.cpp splits it across the 2 parallel slots.
f16 KV drops from ~32 GiB to ~24 GiB, adding OOM headroom.
Docs: sync the 2026-08-18 service swap (frank-default-large on dankclaw,
frank-default-small on dankserver) across AGENTS.md, README.md,
deployment.md, ciru-activefpx-promptforge.md, and index.md. The pi
models.json contextWindow/maxTokens for frank-default-large move
262144 -> 200000 (separate file, outside the repo).
Replace the dead dankserver open-webui (bare container, :3000, pointed at a
dead :9090 backend) with a boot-persistent quadlet on dankclaw:
- CPU image (:main); dankclaw is CPU-only, so the :cuda image is wasted.
- Isolated on slirp4netns; publishes 3005 (3000 is Gitea's HTTP port).
- Default LLM backend is the local vLLM on 52625 (frank-default-large).
- Data persists under /home/dank/Containers/open-webui/data.
- Caddy route now proxies to the published LAN port (the container is not on
Caddy's network).
Also corrects the model/port mapping in the docs: the live state (verified
2026-08-19) is frank-default-large on vLLM :52625 (dankclaw) and
frank-default-small on llama.cpp :7770 (dankserver) — the reverse of the
2026-08-17 note.
Quadlet units are generator-managed; systemctl enable fails on them
('transient or generated'). The [Install] WantedBy=default.target in each
.container file is itself the enablement (is-enabled reports 'generated'), and
with user linger on they start at boot. Drop the enable step; daemon-reload and
start are enough.
Device= in [Container] is a [Volume]-group key; the container device key is
AddDevice= (maps to podman --device). Update the /dev/net/tun device line and
the test seam.
Frank