Full TDD (Q36): red test first, then the minimal implementation.
- tests/homepage_checker.bats: seam = homepage-status.py <services.json>
<out.json>. Asserts an open port is up, a closed port is down, a wip
service is wip (not probed), and the output carries generated_at.
- scripts/homepage-status.py: reads the config, probes each backend (tcp,
5 s timeout), skips wip, writes status.json atomically (temp + rename).
Remaining slices: http probe method, services.json schema, the pages +
theme + toggle, the Caddyfile (validate + two hashes), the git-driven deploy.
- Homepage Design Grilling.md: final decision table (Q1-Q43) + the deltas where
the operator changed the recommendation (per-service probe method, 5s timeout,
TUI panels, toggleable styling, no cursor alteration, no auto-refresh, /docs/
path, user-facing docs, git-driven deploy, full TDD, last-deploy footer).
- Homepage.md: updated to the settled design (per-service probe, 5s, TUI panels,
styling toggle, no auto-refresh, /docs/ workstream, git-driven deploy, TDD).
- pi log.md: 2026-08-23 entry recording the settled design and build order.
Build order is unblocked. No homepage code built yet.
- Homepage.md: rewritten to the settled design (single site, user + /admin/
pages, server-side status.json via a 3-min systemd-timer checker, terminal
TUI theme). Added mermaid diagrams: site structure, status states, checker
flow. Recorded the classification (user vs admin; *arr excluded; LLM admin;
backends status-only) and the documented future shift to client-side probing.
- Homepage Design Grilling.md (new): the decision log — settled Q1-Q5 plus 38
open questions (Q6-Q43) across 10 categories, each with a recommendation.
- docs/index.md: link the grilling page (satisfies the index-links-every-page test).
- report/homepage-grilling-2026-08-22.html: self-contained terminal-TUI progress
report (settled decisions, verified stack baseline, overnight processing, work
threads, the open questions, and how to continue).
Baseline recorded: Dankclaw fully up (10/10); the Dankserver media stack is down
(only Calibre-Web + the LLM are up; no .env on the target) — a separate thread.
No homepage code built yet; the design is pending the operator's answers to Q6-Q43.
The explanatory comment contained an unescaped loop-variable token, which
the local shell (set -u) tried to expand while building the double-quoted
remote script ('unit: unbound variable'). Reword the comment.
The relink loop wrapped the source path in single quotes, which shipped a
literal ${unit} to the remote and created a dangling symlink
(/etc/systemd/system/frank-default-large.service -> .../${unit}.service),
so 'systemctl enable' failed with 'unit does not exist'. The path has no
spaces; unquoting lets ${unit} expand on the remote for both source and
destination.
Operator request (2026-08-22): the primary inference server
(frank-default-large, the 27B CIRU model on Dankclaw) moves to the
standard llama.cpp port 7770. Both LLM services now listen on :7770,
distinguished by host (dankserver vs dankclaw).
Functional changes:
- scripts/frank-default-large.env: LISTEN_PORT 52625 -> 7770
- scripts/serve-frank-large.sh: default LISTEN_PORT 52625 -> 7770
- scripts/deploy.sh: LLM_HEALTH[dankclaw] -> :7770
- open-webui quadlet: OPENAI_API_BASE_URL -> 192.168.0.143:7770
(re-pointed so chat keeps working; data dir preserved)
- scripts/bench-fast-greedy.sh: PORT 52625 -> 7770
Tests updated to the new topology (both LLM endpoints on :7770 by host;
open-webui -> :7770). Docs + pi KB updated; dated historical runbooks
keep their point-in-time ports. models.json (pi client) re-pointed to
:7770 separately, effective on the next client restart.
- docs/index.md: LLM Fleet section (all 34 pages linked), merge note,
See Also now internal
- docs/LLM/index.md: retitled, dangling [[Dankserver]]/[[Dankclaw]]
links fixed, sibling section replaced with internal links
- docs/LLM/deployment.md: rewritten for the unified deploy flow,
manual-update paths point at the merged checkout, incident history
kept intact
- Deployment Flow / Dankclaw Stack / Gitea: stale vllmScripts
references and retired-topology notes updated to the 2026-08-22
state
- Historical docs (frank-redo, PP deployment, FastFlowLM) keep their
dated runbook paths as-is (records of retired topologies)
deploy.sh now deploys both workloads per target in one run:
push -> pull on /home/dank/serverAppConfigs -> podman stack steps
(dankclaw) -> LLM unit relink + daemon-reload + service restart ->
homepage health + LLM health (up to 20 min model-load wait).
Sudo: plain sudo on the fleet hosts (passwordless verified
2026-08-22); FRANK_SUDO_PW covers headless runs elsewhere. The
vllmScripts deploy.sh is superseded.
Units: frank-default-{small,large,npu}.service now reference
/home/dank/serverAppConfigs/scripts/... (the merged checkout).
bench-spec-stack.sh: env file path updated to the merged checkout.
README: merged service table (stack + LLM), structure incl. old/ and
docs/LLM/, unified deploy note, merge note.
.gitignore: merged both files; !scripts/*.env exception for the LLM
tunables (paths and numbers, never credentials); .obsidian/,
__pycache__/, .venv/ from the vllmScripts side.
One repo, one git protocol, one deploy script. Merged from the two
former AGENTS.md files:
- Overview: both workloads, both hosts, the 2026-08-22 merge note,
current LLM fleet topology (frank-default-small :7770 Dankserver,
frank-default-large :52625 Dankclaw)
- Git workflow + testing (serverAppConfigs protocol, test step kept)
- Deployment: the unified deploy.sh (stack + LLM per target)
- Secret rules: no passwords in the repo; FRANK_SUDO_PW covers
headless sudo (the live sudo password is NOT carried forward)
- Deletion & Trash Policy (from the vllmScripts 2026-08-22 commit)
- Benchmarking policy (stop the service before any bench)
- Pi model config, runtime & troubleshooting (v3 CIRU active,
Kairic/DFlash2/FastMTP retired), HIP env naming rule, crash-loop
procedure, operational quirks (passwordless sudo verified
2026-08-22), Dankclaw memory model
New seams (written first, red against the pre-merge tree):
- llm_units.bats: every scripts/*.service unit points at files that
exist in this repo (the merged checkout path), env files included
- llm_secrets.bats: the live sudo password is never tracked (the
vllmScripts AGENTS.md once documented it)
- deployment_flow.bats: deploy.sh must restart the LLM services on
both targets and health-check both endpoints (7770, 52625)
- helpers.bash: has_no_leaked_sudo_pw (two-part assembly, same
pattern as the gitea check)
The LLM fleet serve scripts, systemd units, env tunables, and bench
harnesses join the deploy scripts. The vllmScripts deploy.sh is
removed here; its functionality is folded into the unified
scripts/deploy.sh in a follow-up commit.
bench-spec-stack.sh gains the executable bit (required by the
scripts_syntax test seam).
The LLM fleet documentation (deployment runbook, model runbooks,
benchmarks, research archive, retired-topology history) joins the
project knowledge base as docs/LLM/. Wikilinks resolve by filename,
so no link updates are needed for the move.
The vllmScripts repository (LLM inference fleet: serve scripts, systemd
units, env tunables, deployment runbook, research docs) is merged into
this repository. Its full history is grafted under llm/ with a
two-parent merge commit so no commit is lost.
A follow-up commit reorganizes llm/ into the repo layout:
llm/docs -> docs/LLM, llm/scripts -> scripts, llm/old -> old.
Explicit rule: stage deletions in a dated ~/.AGENTTRASH/YYYYMMDD-topic
directory, list sizes, and wait for user approval before purging. rm,
rm -rf, and git clean -f forbidden without explicit permission. Applies
to every host.
LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=0 unconditionally. FASTGREEDY=1 now
means: reasoning off, deterministic sampler defaults, compatible with
all non-greedy requests (temp != 0, penalties, reasoning budget). This
is the card's 'compatible mode' profile.
The bridge gateway (10.89.3.1) lives in the rootless netns; llama-server
runs in the host netns, so the gateway refuses. 192.168.0.143:52625
reaches it through the rootless NAT (verified from the container).
Same LAN-IP pattern the doc already prescribes for the Dankserver
backend.
A 'caddy reload' delivered over the admin API cannot apply a config
that disables the admin API -- the fix cannot ride the API it is meant
to enable (2026-08-22: admin off sat in the running config across
deploys). Track the reload result; on failure, restart the container
so it reads the rendered file on start.
Reasoning off, argmax fastpath on, deterministic sampler. The fast
greedy path rejects non-greedy requests with 400; chat clients must
send temperature 0.
The Caddyfile had 'admin off', which disables Caddy's config file
watcher. Rendered Caddyfile changes never loaded until the container
was recreated, so the double-hash basicauth fix sat on disk unapplied
across deploys (2026-08-22). Admin binds to 127.0.0.1:2019 inside the
container and the port is unpublished, so nothing outside reaches it.
deploy.sh now issues an explicit 'caddy reload' (retry loop) after
'up -d', with the file watcher as backstop.
basicauth (2026-08-22 incident):
- the Caddyfile template carried its own $2a$14$ prefix and deploy.sh
substituted the full hash (prefix included), rendering an invalid
double-prefixed hash. Every basicauth vhost rejected every password
with 401 and Caddy logged nothing.
- template is now a bare REPLACE_WITH_HASH token; tests pin the render
contract (no prefix in template; rendered line is exactly 'dank <hash>').
open-webui (2026-08-22 incident):
- its slirp4netns helper process died, leaving the container running
with only 'lo' in its netns: port 3005 accepted TCP and reset every
request, so chat served empty responses for ~2 days, unlogged.
- move to a pinned bridge network (systemd-open-webui, 10.89.3.0/24,
gw 10.89.3.1) — no per-container helper to die.
- LLM endpoint reached via the gateway (host.containers.internal is
slirp-only): OPENAI_API_BASE_URL=http://10.89.3.1:52625.
- add a quadlet healthcheck: /health (app) + non-loopback interface
(network liveness; the dead-tap state leaves only 'lo').
- deploy script now copies the .network file too.
- docs updated (OpenWebUI History section, Caddy basic-auth contract).
buildah can serve a stale COPY --from layer (cache keyed on stage
name), so a Containerfile edit silently produced an unchanged image.
The image now carries a cfhash label; a mismatch forces a --no-cache
rebuild.
- alpine's /bin/busybox is musl-dynamic and cannot execute in the
distroless loki image (missing loader); busybox-static links in
- alloy 1.x refuses to configure the ui service from a config block;
the default UI already listens on 127.0.0.1:12345 where the
healthcheck probes
- alloy 1.x renamed the server component to ui; the old block failed the
initial load and the container crash-looped
- grafana/loki is distroless and grafana/alloy lacks wget, so their
healthchecks could never run; local images add a static busybox from
alpine for that one command
- deploy.sh builds all three local images if missing
- tests: 3 new seams
Measured the rootless mapping: container root IS the invoking user
(dank, owner of the /opt data dirs); the images' default users (loki
10001, prometheus nobody, grafana 472) map to subuids that own nothing.
1000:1000 was the wrong direction.
Container root maps to a subuid that owns nothing under /opt, so the
data-dir writes fail. Container uid 1000 is the invoking user in
rootless podman. Fixes the awk service-block seam to use regex match.
- loki: recent releases require compactor.delete_request_store when
retention is enabled (was crash-looping on start)
- prometheus: scrape targets carried legacy dankclaw.local names with
URL schemes (invalid); point at published LAN ports and the gitea
quadlet's published port instead
- grafana: GF_SERVER_ROOT_URL to the canonical HTTPS name
- tests: 3 new seams for the deployed observability configs
- Caddyfile: drop http:// site prefixes, global acme_dns porkbun block,
HTTPS redirects on the bare-IP and legacy vhosts
- new dankclaw/caddy/Containerfile: xcaddy build with caddy-dns/porkbun
- compose: caddy uses franksdank/caddy-porkbun:latest + Porkbun env vars
- deploy.sh: build the image on first deploy if missing
- .env.example: rename PORKBUN_API_SECRET -> PORKBUN_API_SECRET_KEY
(the name the caddy-dns/porkbun module expects)
- tests: 7 new caddyfile/compose seams for the TLS config
Local overrides to the pinned runner for production chat:
- --reasoning on --reasoning-format deepseek (thinking to
message.reasoning_content, matching the retired v3 service so the
fleet's reasoning clients work)
- --reasoning-budget -1 (unrestricted thinking)
- --chat-template-kwargs '{"preserve_thinking":"true"}' (preserve
thinking tokens in the conversation history across turns)
- batch 4096 / ubatch 2048 (larger than the runner's 2048/512; unified
memory has headroom)
The Kairic IU4 lane (--kairic-edge) and the MTP draft fast path are
unaffected.
Frank