The vllmScripts repository (LLM inference fleet: serve scripts, systemd
units, env tunables, deployment runbook, research docs) is merged into
this repository. Its full history is grafted under llm/ with a
two-parent merge commit so no commit is lost.
A follow-up commit reorganizes llm/ into the repo layout:
llm/docs -> docs/LLM, llm/scripts -> scripts, llm/old -> old.
Explicit rule: stage deletions in a dated ~/.AGENTTRASH/YYYYMMDD-topic
directory, list sizes, and wait for user approval before purging. rm,
rm -rf, and git clean -f forbidden without explicit permission. Applies
to every host.
LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=0 unconditionally. FASTGREEDY=1 now
means: reasoning off, deterministic sampler defaults, compatible with
all non-greedy requests (temp != 0, penalties, reasoning budget). This
is the card's 'compatible mode' profile.
The bridge gateway (10.89.3.1) lives in the rootless netns; llama-server
runs in the host netns, so the gateway refuses. 192.168.0.143:52625
reaches it through the rootless NAT (verified from the container).
Same LAN-IP pattern the doc already prescribes for the Dankserver
backend.
A 'caddy reload' delivered over the admin API cannot apply a config
that disables the admin API -- the fix cannot ride the API it is meant
to enable (2026-08-22: admin off sat in the running config across
deploys). Track the reload result; on failure, restart the container
so it reads the rendered file on start.
Reasoning off, argmax fastpath on, deterministic sampler. The fast
greedy path rejects non-greedy requests with 400; chat clients must
send temperature 0.
The Caddyfile had 'admin off', which disables Caddy's config file
watcher. Rendered Caddyfile changes never loaded until the container
was recreated, so the double-hash basicauth fix sat on disk unapplied
across deploys (2026-08-22). Admin binds to 127.0.0.1:2019 inside the
container and the port is unpublished, so nothing outside reaches it.
deploy.sh now issues an explicit 'caddy reload' (retry loop) after
'up -d', with the file watcher as backstop.
basicauth (2026-08-22 incident):
- the Caddyfile template carried its own $2a$14$ prefix and deploy.sh
substituted the full hash (prefix included), rendering an invalid
double-prefixed hash. Every basicauth vhost rejected every password
with 401 and Caddy logged nothing.
- template is now a bare REPLACE_WITH_HASH token; tests pin the render
contract (no prefix in template; rendered line is exactly 'dank <hash>').
open-webui (2026-08-22 incident):
- its slirp4netns helper process died, leaving the container running
with only 'lo' in its netns: port 3005 accepted TCP and reset every
request, so chat served empty responses for ~2 days, unlogged.
- move to a pinned bridge network (systemd-open-webui, 10.89.3.0/24,
gw 10.89.3.1) — no per-container helper to die.
- LLM endpoint reached via the gateway (host.containers.internal is
slirp-only): OPENAI_API_BASE_URL=http://10.89.3.1:52625.
- add a quadlet healthcheck: /health (app) + non-loopback interface
(network liveness; the dead-tap state leaves only 'lo').
- deploy script now copies the .network file too.
- docs updated (OpenWebUI History section, Caddy basic-auth contract).
buildah can serve a stale COPY --from layer (cache keyed on stage
name), so a Containerfile edit silently produced an unchanged image.
The image now carries a cfhash label; a mismatch forces a --no-cache
rebuild.
- alpine's /bin/busybox is musl-dynamic and cannot execute in the
distroless loki image (missing loader); busybox-static links in
- alloy 1.x refuses to configure the ui service from a config block;
the default UI already listens on 127.0.0.1:12345 where the
healthcheck probes
- alloy 1.x renamed the server component to ui; the old block failed the
initial load and the container crash-looped
- grafana/loki is distroless and grafana/alloy lacks wget, so their
healthchecks could never run; local images add a static busybox from
alpine for that one command
- deploy.sh builds all three local images if missing
- tests: 3 new seams
Measured the rootless mapping: container root IS the invoking user
(dank, owner of the /opt data dirs); the images' default users (loki
10001, prometheus nobody, grafana 472) map to subuids that own nothing.
1000:1000 was the wrong direction.
Container root maps to a subuid that owns nothing under /opt, so the
data-dir writes fail. Container uid 1000 is the invoking user in
rootless podman. Fixes the awk service-block seam to use regex match.
- loki: recent releases require compactor.delete_request_store when
retention is enabled (was crash-looping on start)
- prometheus: scrape targets carried legacy dankclaw.local names with
URL schemes (invalid); point at published LAN ports and the gitea
quadlet's published port instead
- grafana: GF_SERVER_ROOT_URL to the canonical HTTPS name
- tests: 3 new seams for the deployed observability configs
- Caddyfile: drop http:// site prefixes, global acme_dns porkbun block,
HTTPS redirects on the bare-IP and legacy vhosts
- new dankclaw/caddy/Containerfile: xcaddy build with caddy-dns/porkbun
- compose: caddy uses franksdank/caddy-porkbun:latest + Porkbun env vars
- deploy.sh: build the image on first deploy if missing
- .env.example: rename PORKBUN_API_SECRET -> PORKBUN_API_SECRET_KEY
(the name the caddy-dns/porkbun module expects)
- tests: 7 new caddyfile/compose seams for the TLS config
Local overrides to the pinned runner for production chat:
- --reasoning on --reasoning-format deepseek (thinking to
message.reasoning_content, matching the retired v3 service so the
fleet's reasoning clients work)
- --reasoning-budget -1 (unrestricted thinking)
- --chat-template-kwargs '{"preserve_thinking":"true"}' (preserve
thinking tokens in the conversation history across turns)
- batch 4096 / ubatch 2048 (larger than the runner's 2048/512; unified
memory has headroom)
The Kairic IU4 lane (--kairic-edge) and the MTP draft fast path are
unaffected.
The pinned runner sets LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=1, but the
exact-Q8 argmax fast path fails closed on any non-greedy request. The
model card's own chat instructions override sampling per request with
Qwen live values (temp 0.7, top-p 0.8, top-k 20, min-p 0,
presence_penalty 1.5), which the fast path rejects (400). Production
chat forces the fast path to 0; the Kairic IU4 lane (--kairic-edge) and
the MTP draft fast path are unaffected.
Serves jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (four-artifact payload) with
the qualified pinned-runner profile: 262K ctx, 1 slot, b2048/ub512,
t16/tb32, f16 KV, 32 ctx checkpoints, 8 GiB prompt cache with idle-slot
persistence, native MTP4, deterministic sampler, reasoning off.
The serve script replicates the release pinned runner verbatim; the only
differences are safe bind vars (no bare HOST env var; HIP collision rule)
and this machine's split 7.15 SDK LD_LIBRARY_PATH layout.
Rollback: revert this commit and redeploy; v3 model files and the v3
binary are untouched.
Per user direction, Caddy runs only on dankclaw. It fronts every
service on both hosts: local services by container name, quadlet
services by published LAN port, dankserver services by published LAN
port (192.168.0.235:PORT).
- dankclaw/Caddyfile: merged vhost list (was: two Caddyfiles, one per
host). gitea's single vhost moves in from the retired gitea-caddy
quadlet (option B). Upstreams fixed to live reality: gitea :3000,
search :8081 (gluetun), chat :3005 (open-webui quadlet), media
services via 192.168.0.235. Retired: llms (open-webui on dankserver
is removed; chat is its successor), the dankserver homepage (302 to
the shared one).
- dankserver/Caddyfile, dankclaw/gitea/Caddyfile: deleted (git rm).
- dnsmasq addn-hosts: every name -> 192.168.0.143 (the one Caddy
host); no .235 records, no llms.
- compose.yaml: removed the openwebui/postgres/gitea services (they
are quadlets now); caddy service mounts the rendered Caddyfile from
/home/dank/Containers/caddy, healthcheck probes the bare-IP
catch-all (302).
- deploy.sh: per-target logic (dankserver = pull-only, health via the
shared Caddy at 192.168.0.143); renders CADDY_BASICAUTH_HASH from
the host .env into the mounted Caddyfile.
- .env.example: CADDY_BASICAUTH_HASH on dankclaw; Porkbun block
dropped from dankserver (its Caddy is gone); SEARXNG_BASE_URL to
the flat name; stale OPENAI_* block dropped (quadlet is
self-contained).
- tests: caddyfile.bats now asserts the single-Caddyfile invariant;
dnsmasq.bats cross-checks vhosts -> .143 and the retired names.
/opt is root-owned; the established data-location pattern (see
docs/Architecture/Storage Policy.md, open-webui quadlet) is
/home/dank/Containers/<service>.
Phase 1 of the local-DNS-name work. Users type a name, not a port.
- dnsmasq quadlet on dankclaw (rootless, slirp4netns, port 53 published):
serves the flat name scheme from addn-hosts, upstream 1.1.1.1/1.0.0.1.
Image built from the in-repo Containerfile (localhost/dnsmasq:latest).
- Caddyfiles (both hosts): all *.dankclaw.local / *.dankserver.local vhosts
renamed to flat <service>.franksdank.cloud. Explicit http:// site
addresses for now; TLS (wildcard LE via DNS-01/Porkbun) lands in phase 2.
- Drift reconciliation (live /etc/caddy/Caddyfile on dankserver differs
from the repo): added the live-only llms.franksdank.cloud vhost
(OpenWebUI on :3000). Dropped the legacy search vhost from dankserver
(the name belongs to dankclaw's gluetun-routed SearXNG; the legacy
instance stays reachable at 192.168.0.235:8081).
- Homepage service links updated to the new names.
- New test seams (TDD): tests/dnsmasq.bats (quadlet boot-persistence,
port 53, conf/record validity, Caddyfile<->addn-hosts cross-check on
the right host IPs); caddyfile.bats naming-convention tests updated.
LAN clients get this DNS from the SURFboard G36 (DNS Relay -> statically
configured -> 192.168.0.143); the router change is a pending user step.
Real tokens live only in the host .env files (git-ignored):
- /home/dank/serverAppConfigs/dankclaw/.env
- /home/dank/serverAppConfigs/dankserver/.env
Token is scoped to the franksdank.cloud zone (DNS read/write).
Frank