The homepage deploy's mkdir/cp/write of /srv/homepage ran as the deploy user
and hit 'Permission denied' (/srv is root-owned). Prefix them with the sudo
wrapper, write deployed.js via a temp file + sudo cp, and chmod the tree
a+rX so Caddy can serve it. Run the one-off checker as root too (it writes
status.json into /srv/homepage).
A missing CADDY_*_HASH in the .env made the 'grep | tail | cut' pipeline
return non-zero, which aborted the deploy under 'set -e' before it could warn
and keep the placeholder. Add '|| true' so an absent hash warns instead of
aborting. (The homepage deploy aborted here because CADDY_ADMIN_HASH was not
in the repo-root .env the deploy reads.)
- Homepage.md: status is now 'built, pending deploy' (all assets + checker +
deploy are in the repo, 112 tests green). To go live: add CADDY_ADMIN_HASH
to the target .env, then deploy.sh.
- Homepage Design Grilling.md: the build is complete (all 6 slices, full TDD);
the only remaining step is the operator adding the admin hash + deploying.
- pi log.md: build-complete entry.
Red tests first.
- tests/homepage_deploy.bats: the deploy ships the homepage (assets to
/srv/homepage, the checker timer, both Caddy hashes, deployed.js) and is
git-driven (pulls main); the checker runs as a systemd timer.
- scripts/homepage-status.service / .timer: the checker as a oneshot service
+ a 3-minute timer (Q2/Q35).
- scripts/deploy.sh: the Caddyfile render now substitutes BOTH hashes
(REPLACE_WITH_HASH <- CADDY_BASICAUTH_HASH, REPLACE_WITH_ADMIN_HASH <-
CADDY_ADMIN_HASH). A new homepage step copies the assets to /srv/homepage,
generates deployed.js from the deployed commit's datetime (Q43), installs
the checker timer, and runs the checker once so status.json exists at once.
- tests/llm_units.bats: scoped the three LLM-unit tests to
frank-default-*.service (the homepage unit is a different pattern: it
installs to /usr/local/bin and runs from /srv at deploy time).
Red tests first.
- tests/homepage_caddy.bats: the homepage site block carries a root hash and
a distinct /admin/-scoped admin hash (Q18/Q38), and serves /srv/homepage.
- dankclaw/Caddyfile: the homepage block now uses named matchers -
@user (not /admin/*) with the shared hash, @admin (/admin/*) with the
operator-only hash (REPLACE_WITH_ADMIN_HASH, rendered from CADDY_ADMIN_HASH
at deploy). Both placeholders are distinct.
Red tests first, then the pages.
- tests/homepage_pages.bats: the pages reference the data + theme; no
auto-refresh (Q41); no cursor alteration (Q28); the styling toggle strips
to bare HTML (Q27); the footer shows the last master deployment datetime
(Q43).
- index.html / admin.html: the user and admin pages. A grid of TUI panels,
a summary line, the bare toggle, the deployed-at footer.
- app.js: reads status.json + services.json once (no auto-refresh), renders
one panel per service for the page's category, the toggle, the footer.
- style.css: the terminal TUI theme (green-on-black, phosphor palette,
scanlines) + a ?bare=1 mode that strips to unstyled HTML. No custom cursor.
Functionally verified with a Node DOM shim: 9 user panels, correct summary,
correct footer datetime.
Red tests first (the old services.json was an object, not the new list schema).
- tests/homepage_config.bats: the config is a JSON list; every entry has
name/category/probe; category is user|admin; method is tcp|http; a
showLink entry has a url and a status-only entry does not.
- shared/homepage/services.json: the real fleet. User: SearXNG, ComfyUI,
OpenWebUI, Gitea, Calibre-Web, RomM, Copyparty, Plex, Jellyfin. Admin:
Hermes, Grafana, Prometheus, Loki, node-exporter, alloy, Caddy, dnsmasq,
Gluetun (wip - a tunnel has no simple local port), the two LLMs, mariadb,
valkey, Postgres. Ports confirmed from the repo config (compose/quadlet).
docs fields are empty for now (the /docs/ workstream fills them as each
service goes active).
Red test first: a raw TCP listener that never speaks HTTP must read down
under an http probe (up under tcp), while a real HTTP server reads up. This
test caught a real hang — the backgrounded listener inherited the test
runner's stdout pipe and held it open; fixed by redirecting its output and
killing both listeners in teardown.
scripts/homepage-status.py: probe() now dispatches on method. http does an
urlopen (any HTTP status = up; no connection = down); tcp connects.
The checker is now complete: tcp + http + wip + atomic write + generated_at.
Remaining slices: services.json schema, the pages + theme + toggle, the
Caddyfile (validate + two hashes), the git-driven deploy.
Full TDD (Q36): red test first, then the minimal implementation.
- tests/homepage_checker.bats: seam = homepage-status.py <services.json>
<out.json>. Asserts an open port is up, a closed port is down, a wip
service is wip (not probed), and the output carries generated_at.
- scripts/homepage-status.py: reads the config, probes each backend (tcp,
5 s timeout), skips wip, writes status.json atomically (temp + rename).
Remaining slices: http probe method, services.json schema, the pages +
theme + toggle, the Caddyfile (validate + two hashes), the git-driven deploy.
- Homepage Design Grilling.md: final decision table (Q1-Q43) + the deltas where
the operator changed the recommendation (per-service probe method, 5s timeout,
TUI panels, toggleable styling, no cursor alteration, no auto-refresh, /docs/
path, user-facing docs, git-driven deploy, full TDD, last-deploy footer).
- Homepage.md: updated to the settled design (per-service probe, 5s, TUI panels,
styling toggle, no auto-refresh, /docs/ workstream, git-driven deploy, TDD).
- pi log.md: 2026-08-23 entry recording the settled design and build order.
Build order is unblocked. No homepage code built yet.
- Homepage.md: rewritten to the settled design (single site, user + /admin/
pages, server-side status.json via a 3-min systemd-timer checker, terminal
TUI theme). Added mermaid diagrams: site structure, status states, checker
flow. Recorded the classification (user vs admin; *arr excluded; LLM admin;
backends status-only) and the documented future shift to client-side probing.
- Homepage Design Grilling.md (new): the decision log — settled Q1-Q5 plus 38
open questions (Q6-Q43) across 10 categories, each with a recommendation.
- docs/index.md: link the grilling page (satisfies the index-links-every-page test).
- report/homepage-grilling-2026-08-22.html: self-contained terminal-TUI progress
report (settled decisions, verified stack baseline, overnight processing, work
threads, the open questions, and how to continue).
Baseline recorded: Dankclaw fully up (10/10); the Dankserver media stack is down
(only Calibre-Web + the LLM are up; no .env on the target) — a separate thread.
No homepage code built yet; the design is pending the operator's answers to Q6-Q43.
The explanatory comment contained an unescaped loop-variable token, which
the local shell (set -u) tried to expand while building the double-quoted
remote script ('unit: unbound variable'). Reword the comment.
The relink loop wrapped the source path in single quotes, which shipped a
literal ${unit} to the remote and created a dangling symlink
(/etc/systemd/system/frank-default-large.service -> .../${unit}.service),
so 'systemctl enable' failed with 'unit does not exist'. The path has no
spaces; unquoting lets ${unit} expand on the remote for both source and
destination.
Operator request (2026-08-22): the primary inference server
(frank-default-large, the 27B CIRU model on Dankclaw) moves to the
standard llama.cpp port 7770. Both LLM services now listen on :7770,
distinguished by host (dankserver vs dankclaw).
Functional changes:
- scripts/frank-default-large.env: LISTEN_PORT 52625 -> 7770
- scripts/serve-frank-large.sh: default LISTEN_PORT 52625 -> 7770
- scripts/deploy.sh: LLM_HEALTH[dankclaw] -> :7770
- open-webui quadlet: OPENAI_API_BASE_URL -> 192.168.0.143:7770
(re-pointed so chat keeps working; data dir preserved)
- scripts/bench-fast-greedy.sh: PORT 52625 -> 7770
Tests updated to the new topology (both LLM endpoints on :7770 by host;
open-webui -> :7770). Docs + pi KB updated; dated historical runbooks
keep their point-in-time ports. models.json (pi client) re-pointed to
:7770 separately, effective on the next client restart.
- docs/index.md: LLM Fleet section (all 34 pages linked), merge note,
See Also now internal
- docs/LLM/index.md: retitled, dangling [[Dankserver]]/[[Dankclaw]]
links fixed, sibling section replaced with internal links
- docs/LLM/deployment.md: rewritten for the unified deploy flow,
manual-update paths point at the merged checkout, incident history
kept intact
- Deployment Flow / Dankclaw Stack / Gitea: stale vllmScripts
references and retired-topology notes updated to the 2026-08-22
state
- Historical docs (frank-redo, PP deployment, FastFlowLM) keep their
dated runbook paths as-is (records of retired topologies)
deploy.sh now deploys both workloads per target in one run:
push -> pull on /home/dank/serverAppConfigs -> podman stack steps
(dankclaw) -> LLM unit relink + daemon-reload + service restart ->
homepage health + LLM health (up to 20 min model-load wait).
Sudo: plain sudo on the fleet hosts (passwordless verified
2026-08-22); FRANK_SUDO_PW covers headless runs elsewhere. The
vllmScripts deploy.sh is superseded.
Units: frank-default-{small,large,npu}.service now reference
/home/dank/serverAppConfigs/scripts/... (the merged checkout).
bench-spec-stack.sh: env file path updated to the merged checkout.
README: merged service table (stack + LLM), structure incl. old/ and
docs/LLM/, unified deploy note, merge note.
.gitignore: merged both files; !scripts/*.env exception for the LLM
tunables (paths and numbers, never credentials); .obsidian/,
__pycache__/, .venv/ from the vllmScripts side.
One repo, one git protocol, one deploy script. Merged from the two
former AGENTS.md files:
- Overview: both workloads, both hosts, the 2026-08-22 merge note,
current LLM fleet topology (frank-default-small :7770 Dankserver,
frank-default-large :52625 Dankclaw)
- Git workflow + testing (serverAppConfigs protocol, test step kept)
- Deployment: the unified deploy.sh (stack + LLM per target)
- Secret rules: no passwords in the repo; FRANK_SUDO_PW covers
headless sudo (the live sudo password is NOT carried forward)
- Deletion & Trash Policy (from the vllmScripts 2026-08-22 commit)
- Benchmarking policy (stop the service before any bench)
- Pi model config, runtime & troubleshooting (v3 CIRU active,
Kairic/DFlash2/FastMTP retired), HIP env naming rule, crash-loop
procedure, operational quirks (passwordless sudo verified
2026-08-22), Dankclaw memory model
New seams (written first, red against the pre-merge tree):
- llm_units.bats: every scripts/*.service unit points at files that
exist in this repo (the merged checkout path), env files included
- llm_secrets.bats: the live sudo password is never tracked (the
vllmScripts AGENTS.md once documented it)
- deployment_flow.bats: deploy.sh must restart the LLM services on
both targets and health-check both endpoints (7770, 52625)
- helpers.bash: has_no_leaked_sudo_pw (two-part assembly, same
pattern as the gitea check)
The LLM fleet serve scripts, systemd units, env tunables, and bench
harnesses join the deploy scripts. The vllmScripts deploy.sh is
removed here; its functionality is folded into the unified
scripts/deploy.sh in a follow-up commit.
bench-spec-stack.sh gains the executable bit (required by the
scripts_syntax test seam).
The LLM fleet documentation (deployment runbook, model runbooks,
benchmarks, research archive, retired-topology history) joins the
project knowledge base as docs/LLM/. Wikilinks resolve by filename,
so no link updates are needed for the move.
The vllmScripts repository (LLM inference fleet: serve scripts, systemd
units, env tunables, deployment runbook, research docs) is merged into
this repository. Its full history is grafted under llm/ with a
two-parent merge commit so no commit is lost.
A follow-up commit reorganizes llm/ into the repo layout:
llm/docs -> docs/LLM, llm/scripts -> scripts, llm/old -> old.
Explicit rule: stage deletions in a dated ~/.AGENTTRASH/YYYYMMDD-topic
directory, list sizes, and wait for user approval before purging. rm,
rm -rf, and git clean -f forbidden without explicit permission. Applies
to every host.
LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=0 unconditionally. FASTGREEDY=1 now
means: reasoning off, deterministic sampler defaults, compatible with
all non-greedy requests (temp != 0, penalties, reasoning budget). This
is the card's 'compatible mode' profile.
The bridge gateway (10.89.3.1) lives in the rootless netns; llama-server
runs in the host netns, so the gateway refuses. 192.168.0.143:52625
reaches it through the rootless NAT (verified from the container).
Same LAN-IP pattern the doc already prescribes for the Dankserver
backend.
A 'caddy reload' delivered over the admin API cannot apply a config
that disables the admin API -- the fix cannot ride the API it is meant
to enable (2026-08-22: admin off sat in the running config across
deploys). Track the reload result; on failure, restart the container
so it reads the rendered file on start.
Reasoning off, argmax fastpath on, deterministic sampler. The fast
greedy path rejects non-greedy requests with 400; chat clients must
send temperature 0.
The Caddyfile had 'admin off', which disables Caddy's config file
watcher. Rendered Caddyfile changes never loaded until the container
was recreated, so the double-hash basicauth fix sat on disk unapplied
across deploys (2026-08-22). Admin binds to 127.0.0.1:2019 inside the
container and the port is unpublished, so nothing outside reaches it.
deploy.sh now issues an explicit 'caddy reload' (retry loop) after
'up -d', with the file watcher as backstop.
basicauth (2026-08-22 incident):
- the Caddyfile template carried its own $2a$14$ prefix and deploy.sh
substituted the full hash (prefix included), rendering an invalid
double-prefixed hash. Every basicauth vhost rejected every password
with 401 and Caddy logged nothing.
- template is now a bare REPLACE_WITH_HASH token; tests pin the render
contract (no prefix in template; rendered line is exactly 'dank <hash>').
open-webui (2026-08-22 incident):
- its slirp4netns helper process died, leaving the container running
with only 'lo' in its netns: port 3005 accepted TCP and reset every
request, so chat served empty responses for ~2 days, unlogged.
- move to a pinned bridge network (systemd-open-webui, 10.89.3.0/24,
gw 10.89.3.1) — no per-container helper to die.
- LLM endpoint reached via the gateway (host.containers.internal is
slirp-only): OPENAI_API_BASE_URL=http://10.89.3.1:52625.
- add a quadlet healthcheck: /health (app) + non-loopback interface
(network liveness; the dead-tap state leaves only 'lo').
- deploy script now copies the .network file too.
- docs updated (OpenWebUI History section, Caddy basic-auth contract).
buildah can serve a stale COPY --from layer (cache keyed on stage
name), so a Containerfile edit silently produced an unchanged image.
The image now carries a cfhash label; a mismatch forces a --no-cache
rebuild.
- alpine's /bin/busybox is musl-dynamic and cannot execute in the
distroless loki image (missing loader); busybox-static links in
- alloy 1.x refuses to configure the ui service from a config block;
the default UI already listens on 127.0.0.1:12345 where the
healthcheck probes
- alloy 1.x renamed the server component to ui; the old block failed the
initial load and the container crash-looped
- grafana/loki is distroless and grafana/alloy lacks wget, so their
healthchecks could never run; local images add a static busybox from
alpine for that one command
- deploy.sh builds all three local images if missing
- tests: 3 new seams
Measured the rootless mapping: container root IS the invoking user
(dank, owner of the /opt data dirs); the images' default users (loki
10001, prometheus nobody, grafana 472) map to subuids that own nothing.
1000:1000 was the wrong direction.
Container root maps to a subuid that owns nothing under /opt, so the
data-dir writes fail. Container uid 1000 is the invoking user in
rootless podman. Fixes the awk service-block seam to use regex match.
- loki: recent releases require compactor.delete_request_store when
retention is enabled (was crash-looping on start)
- prometheus: scrape targets carried legacy dankclaw.local names with
URL schemes (invalid); point at published LAN ports and the gitea
quadlet's published port instead
- grafana: GF_SERVER_ROOT_URL to the canonical HTTPS name
- tests: 3 new seams for the deployed observability configs
Frank