Commit Graph
57 Commits
Author SHA1 Message Date
Frank d3c819741d homepage: add suggested uses to the three LLM model descriptions
frank-large: cybersecurity + debugging (requires Frank's permission)
frank-medium: primary intelligent model (surpasses Claude Sonnet 5 per
  Artificial Analysis) — software engineering, sysadmin, instruction tasks
frank-small: fast/cheap utility model — agentic search, RAG, research
  subagents, translation, roleplay, summarization, small utility tasks

Updated both the status cards (services.json) and the /docs model guide.
2026-09-25 22:32:56 -07:00
Frank 66a5e39899 homepage: add self-serve account link to authentik user-settings flow
Footer link opens the authentik default-user-settings-flow in a new tab,
giving users a self-service entry point to change their password and
manage MFA/credentials without admin intervention.
2026-09-24 20:43:11 -07:00
FrankandClaude 600812d459 docs: update frank-large to stock llama.cpp UD-Q8_K_XL + DFlash2
- AGENTS.md: fleet topology, intro, frank-large section, text-only note,
  benchmarking policy, runbook index (new runbook active; Kairic Edge
  retired as rollback target)
- homepage services.json: frank-large description -> UD-Q8_K_XL, 31.5 GiB,
  DFlash2 spec decode
- homepage services.json: ComfyUI url + showLink (complete prior edit)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-09-21 21:53:53 -07:00
Frank be878008a2 homepage: update LLM descriptions for model changes
frank-large: Qwen3.8-Flash-Next UD-IQ4_XS (87.3 GiB, 200k ctx) →
Qwen3.8-27B IU4/W4A4 (26 GiB, 262k ctx) — the Kairic Edge swap.
frank-medium: Qwen3.8-27B-Uncensored FP8 (~31 GiB) → MXFP4/W4A8
(19.4 GiB) — the MXFP4 checkpoint cut.
frank-small: unchanged (Gemma4-12B QAT).
2026-09-21 20:19:05 -07:00
Frank 2f7aadab88 homepage: add the Hermes dashboard (agent management UI) card
The hermesadmin.franksdank.cloud dashboard (9119 — config, API keys,
sessions) is a distinct admin surface from the web UI client. Add it as a
third AI-section card; probe 9119 (302 login redirect counts as up, verified
on Dankclaw).
2026-09-21 17:37:11 -07:00
Frank af36595117 homepage: link the Pi-hole admin and the Hermes web UI
The admin Infrastructure section carried the retired 'dnsmasq' (probe-only,
no link) — Pi-hole FTL (a dnsmasq fork) replaced it on 2026-08-24. Replace it
with a linked Pi-hole card (pihole.franksdank.cloud, probe 8083 — the 403
basicauth counts as up). The Hermes card was probe-only; add the web UI
client link (hermes.franksdank.cloud). Both probes verified up on Dankclaw.
2026-09-21 17:30:48 -07:00
Frank 2b046a21d0 homepage: AI section to the bottom; model-only AI descriptions
The user page now orders Web Apps, Media Library Apps, AI — the models sit
at the bottom, below the apps people click. The AI card and /docs#models
descriptions drop the infrastructure (runtimes, GPUs, tensor parallelism,
draft heads) in favor of what users care about: the hosted model, its
size, the context/parallel capacity, and what it's good at.
2026-09-21 15:02:14 -07:00
Frank e22b43d589 homepage: sections are horizontal rows, not columns
Each section's panels fill an auto-fill row (as many per row as the
viewport allows) and the sections stack vertically — the side-by-side
column layout was the wrong read of 'render the sections horizontally'.
2026-09-21 14:30:21 -07:00
Frank d6f9b8539a fix(frank-large): load the model as its native 3-way split, not a concatenated file
The first cutover crash-looped the unit with 'invalid split file name':
the fetch script concatenated the three split parts into a single GGUF,
but part 1's header carries ggml.file_count=3, so the byte-correct
concatenation still advertises itself as a split and the loader demands
a -NNNNN-of-NNNNN filename.

- serve-frank-large.sh: -m points at part 1; llama.cpp loads the split
  natively (parts 2-3 auto-discovered in the same directory)
- deploy-remote.sh: LLM_LARGE_MODEL gate path follows
- download-frank-large-unsloth.sh: assembly removed; the parts ARE the
  canonical payload, verified against live Hub LFS oids
- docs: runbook model section + troubleshooting note, AGENTS.md, log
- the redundant 87 GiB assembled file is staged on dankclaw at
  ~/.AGENTTRASH/20260921-frank-large-assembled/
2026-09-21 13:58:54 -07:00
Frank 384747a72f homepage: sectioned panels, WIP open/closed, current model facts, Tautulli card
Both pages now group their panels under section headings rendered by shared
renderSection()/renderPanel() components in app.js (SECTION_ORDER fixes each
page's order: user = AI, Web Apps, Media Library Apps; admin = Media Library
Apps, Observability, AI, Infrastructure). The WIP badge splits into WIP: OPEN
and WIP: CLOSED via the wip value ("open"/"closed"); the checker passes the
value through as wip-open/wip-closed. The frank-large card follows the
unsloth llama.cpp cutover, frank-medium follows the FP8 checkpoint, and the
/docs model guide matches. Tautulli (Plex statistics, 192.168.0.235:8181)
joins the admin page's Media Library Apps section with an open link.
2026-09-21 13:20:15 -07:00
Frank 5c1006dda6 fix(monitoring): scrape node-dankclaw via host.containers.internal
node-exporter is net=host, so its 192.168.0.143:9100 bind is a host bind,
not a published port — the bridge-networked Prometheus container cannot
reach its own host's LAN IP there (netavark DNATs published ports only).
The node-dankclaw target has been red by construction since the
observability cutover while the container reported healthy. Point it at
host.containers.internal:9100 (netavark delivers that to the host LAN IP,
where node-exporter listens) — the same rule frank-large and frank-small
already use. node-dankserver is a different machine and keeps its LAN
address. Update the cross-file contract test and the Targets table.
2026-09-21 11:09:03 -07:00
Frank c42780f9f3 fix(monitoring): llama.cpp reuse ratio, dashboard re-cut for both namespaces, measured prefix-cache granularity
- shared/prometheus/rules/llm-fleet.yml: the llamacpp reuse recording rule
  divided cached by prompt, but llama.cpp's prompt_tokens_total EXCLUDES
  cached tokens (its own HELP line), so the ratio was unbounded — measured
  77 on a fully-warm frank-large, meaning LlmPrefixCacheCollapse could never
  fire on that side. Now cached / (cached + prompt), with the traffic floor
  counting both terms too; the header documents the unit trap. Also fixed the
  LlmEndpointDown description: frank-large's CIRU llama.cpp build DOES export
  the llamacpp:* family (verified live), so the alert covers all three
  endpoints.
- shared/grafana/provisioning/dashboards/llm-fleet.json: re-cut for two
  namespaces. Every panel reads both (the endpoint picker selects); the
  KV-pool-fill panel retired 2026-09-16 returned with
  vllm:kv_cache_usage_perc; the latency p95 panel returned with the vLLM
  histograms (TTFT/ITL/e2e); the alerts panel is scoped to the fleet's
  Llm.* rules; the stale halogen-era description is gone. All 21 panel
  expressions validated against the live endpoints.
- runbook + log + Observability: prefix caching measured, not inferred —
  reuse happens at 1,648-token block granularity (attention block size padded
  to the mamba page size); an identical 13,021-token prompt went 2.30 s cold
  to 0.68 s warm with 9,888 tokens (76 %) reused, while a 2,521-token prompt
  registered zero hits. Rule counts corrected to 6 alerts + 3 recording rules.

./tests/run.sh: 325 pass.
2026-09-20 19:53:14 -07:00
Frank 08f1c7ad09 feat(monitoring,harness): follow frank-medium onto vllm:* metrics and fix red-by-construction targets
The radiance TP=2 cutover changed the metric namespace frank-medium exports
(llamacpp:* -> vllm:*) and left two scrape targets addressing host processes
by an address a container cannot reach.

- shared/prometheus/rules/llm-fleet.yml: two namespaces side by side. Added
  LlmRequestsWaiting, LlmKvCacheSaturated (vllm-only gauges llama.cpp never
  had), the job:llm_vllm_prefix_cache_reuse:rate5m and
  job:llm_spec_accept_length:rate5m recording rules, and re-cut
  LlmPrefixCacheCollapse / LlmSpecDecodeOff to OR both families with
  per-family traffic floors so an idle endpoint cannot fire them. Every
  expression validated against live Prometheus.
- shared/prometheus/prometheus.yml: frank-small -> host.containers.internal:7771
  and frank-large scraped at host.containers.internal:7770. Both are HOST
  processes on Dankclaw and netavark DNATs published container ports only, so
  the LAN-IP targets were red by construction; frank-large's CIRU llama.cpp
  build does export the llamacpp:* family, contra the old comment.
- tests/observability_quadlet.bats: the two seams now encode the new contract
  (per-endpoint reachability by process type; every named series must be on the
  measured list for its engine; engine-independent rules must name both).
- homepage docs.html/services.json: frank-medium sees pictures, frank-large is
  CIRU Orca with a single seat (was halogen w4b, two seats, 200k).
- docs: radiance runbook is ACTIVE (deployed flags, three-arm battery, verified
  capabilities, rollback, replica-era sections labelled); llama.cpp runbook
  carries the SUPERSEDED banner; Dual-R9700-Cooperation states the measured
  verdict; Observability records the namespace split and the DNAT lesson;
  AGENTS.md topology/frank-medium/Model Config/Benchmarking Policy/runbook
  index flipped; both indexes updated; work-log entry added.

Also repaired agent-pass damage: prose prepended above frontmatter openers,
duplicated runbook bodies, a literal tool-call token pasted into a runbook, and
an index destroyed by diff artifacts (restored from HEAD, flips re-applied).
Vault link check: 128 pages, 0 unresolved. ./tests/run.sh: 325 pass.
2026-09-20 19:35:56 -07:00
Frank c2116658f8 feat(frank-medium): cut the endpoint over to radiance TP=2
The conversation battery (scripts/bench-conversation.py, salted, same 10
prompts, 2026-09-20) measured three arms on the two R9700s:

  arm I  llama.cpp layer split 32:33 + DFlash2, Q8_0  geo decode  45.7 t/s, TTFT 0.72 s, burst-4  71.7 t/s
  arm L  llama.cpp tensor split 1:1 (RCCL), no drafter geo decode 30.5 t/s, TTFT 0.71 s, burst-4  54.3 t/s
  arm R  radiance vLLM TP=2 + DFlash2-FP8, MXFP4      geo decode  95.8 t/s, TTFT 0.21 s, burst-4 259.4 t/s

Radiance wins every axis, so it takes the alias. The board swap is what made
arm R possible: TP=2 was archived on 2026-09-16 because the second card sat
behind B650 chipset bridges with AtomicOpsCap: Routing-, and the ProArt
B850-Creator put both cards on CPU GPP root ports (Gen5 x8, Routing+, ReqEn+).
The R4D one-shot all-reduce needs no RCCL collective anyway.

  - un-retire dankserver/frank-medium/: the TP=2 quadlet (banner rewritten —
    it is the deployed shape now), radiance-entrypoint.sh, the healthcheck.
    The replica pair and its affinity proxy stay archived; one process owns
    both cards now.
  - retire scripts/frank-medium.{service,env} + serve-frank-medium.sh and
    bench-spec-ab.py to scripts/.AGENTTRASH/20260920-llama-layersplit/ as the
    rollback path. build-llama-rdna4.sh and download-frank-medium-gguf.sh
    stay: they are the tooling that rollback needs.
  - deploy-remote.sh: deploy_llm_medium now gates on the image (pull once,
    re-tag to the localhost name the quadlet resolves), the MXFP4 checkpoint,
    the drafter, the pinned patch clone and the patched libr4d; retires the
    llama root unit into the home trash instead of relinking it; then
    install_quadlet + wait_container_healthy 1800s (first start compiles).
    retire_vllm_medium_quadlets becomes retire_vllm_replica_quadlets — the
    a/b/proxy trio only, since frank-medium is installed again.
  - planner + test-changed map dankserver/frank-medium/** to frank-medium
    and its seam; the scripts-trio mapping is gone with the files.
  - tests/frank_medium_unit.bats rewritten for the container contract: boot
    persistence of a GENERATED unit (no enable on this host's v1 generator),
    TP=2 with the atomics-era NCCL workarounds kept out, the DFlash2 depth vs
    the MXFP4 decode-kernel row band, the KV pin's coupling to the shape it
    was calibrated at plus pool/VRAM arithmetic, the endpoint/tool/reasoning
    contract, and the archive invariants.
  - homepage card + docs updated, and they were stale twice over: they still
    described the ajgazin Swift 35:30 / 393k payload, not the deployed shape.

Known regression, documented in the unit: vision. The MXFP4 checkpoint is
language_model_only and the launcher loads no projector, so images are a 400
here; the harness's vision role already routes to frank-small.
2026-09-20 18:15:27 -07:00
Frank 32751c02d5 homepage: live model info on the fleet cards, user docs page at /docs
- services.json: frank-large/medium/small move from admin-only to the user
  page with current descriptions (halogen w4b iGPU; 35:30 layer-split MTP
  27B on the R9700s; Gemma4-12B QAT vision path on the 3090) and docs
  links to /docs#models
- docs.html: static user documentation (access chain, model guide, media
  how-tos, troubleshooting); served at /docs via the post-auth homepage
  listener; shipped by deploy_homepage; footer + index link to it
- tests: pages seam asserts the docs contract (page, link, Caddy route,
  deployer copy)
- vault: Homepage.md synced (docs landing shipped, cards reclassified)
2026-09-17 23:25:50 -07:00
Frank f73fd43027 relay: bind the host LAN IP, admit the podman bridge subnet only
169.254.1.2 is not a local address on this host (netavark runs its
bridges in aardvark-dns' netns), so the relay cannot bind it — and a
loopback bind refuses container traffic. Probe-socket proof: a host
socket on 192.168.0.143:18094 answers the container's 169.254.1.2:18094
dial while the LAN IP refuses from the container. The relay binds
192.168.0.143:18093 and the deployer installs an iptables INPUT pair
(ACCEPT 10.89.0.0/16, DROP rest) so the unauthenticated endpoint stays
off the LAN.
2026-09-17 13:52:20 -07:00
Frank 8f747a2104 relay: bind 169.254.1.2 (host.containers.internal), not loopback
A socket bound to 127.0.0.1 refuses container traffic that arrives via
the podman bridge: the first live webhook failed with 'dial tcp
169.254.1.2:18093: connection refused'. The relay now binds the
host-gateway endpoint — link-local to the podman bridge, off the LAN —
and the deployer's health probe follows it there.
2026-09-17 13:26:24 -07:00
Frank 590853c133 alerting: Alertmanager + Signal relay deliver firing rules to Frank's phone
- dankclaw/observability/quadlet/alertmanager.container: loopback :9093,
  config from shared/prometheus/alertmanager.yml, family conventions
- shared/prometheus/alertmanager.yml: single signal receiver, group_by
  alertname, 30s/5m/4h intervals, webhook to host.containers.internal:18093
- scripts/signal-relay.py + .service: host-side user unit, loopback :18093,
  15-min dedupe, sends via signal-cli account 318311
- prometheus.yml: alerting block to alertmanager:9093; stale comments fixed
- llm-fleet.yml: delivery header; LlmSpecDecodeOff annotation now drafter-agnostic
- deploy.sh/deploy-remote.sh/deploy-plan.py: alertmanager service, config
  copy, relay install, planner routing for config + relay files
- homepage: alertmanager status card (loopback probe)
- tests: alertmanager in UNITS/loopback/health/PAIRS/planner + five new
  delivery-path seams (webhook<->relay contract, prometheus target, relay
  unit, signal-cli account, planner routing)
- docs: Alertmanager.md component page, Observability.md, Prometheus.md,
  index.md, AGENTS.md delivery line
2026-09-17 13:09:59 -07:00
Frank dc76dc5dc4 frank-medium: serve the whole 262144-token window, and repoint every signal to llama.cpp
Context: 262144 was measured before it was deployed -- 24.72 GB on GPU 0 and
29.14 GB on GPU 1 of 34.21 GB each, ~5 GB of headroom on the busier card, and a
140,069-token prompt prefilled through the single slot in 134 s to prove the
window is real. The 114688 wall was the vLLM predecessor's arithmetic, not this
runtime's.

Throughput, measured through the service: 1.0-1.4k tok/s prefill from 4k to
140k, 43.8 tok/s decode on a 512-token generation, 41-71 % DFlash2 acceptance,
and the prefix cache re-prefilling 4 of 998 tokens on a repeated prompt.

The signals move with the runtime, so the fleet's rules and dashboard are
re-cut rather than renamed: vllm: series do not exist any more, and two of the
old rules watched things llama.cpp does not export at all (KV-pool fill, p95
latency). Those are retired with the reason written down, and LlmSpecDecodeOff
takes one of the slots -- a lost drafter is the failure no client can see. The
observability seam now fails if any rule names a series the endpoint does not
export, which is what would have caught the stale vllm: expressions.

omp models.yml (outside the repo) is updated in the same pass: contextWindow
262144, and the full reasoning-effort map, because Qwen3.8's template answers
500 on 'high' -- verified against the live endpoint, low/medium/xhigh answer
200. Subagents run at high.
2026-09-16 15:29:40 -07:00
Frank ab51d61198 docs: frank-medium is llama.cpp with DFlash2 on both cards
The runbook for the new deployment, plus the sweep of everything that still
described the vllm-radiance replica pair: the vault index and its ACTIVE
marker, the superseded banner on the radiance page, the pointer from the
ROCmFPX page, the homepage card, the Grafana dashboard description and the
observability page's scrape table.

The new page carries the two facts a future agent needs and the flags do not
say: llama.cpp's layer split is what lets BOTH cards serve ONE model without
any collective (vLLM's PP handoff is an RCCL send/recv, which is the primitive
this host cannot dispatch), and the rocBLAS Tensile-path trap that makes a
healthy-looking build fail on the first request.
2026-09-16 15:15:02 -07:00
Frank 79cca19919 observability: the LLM surfaces follow the two-endpoint split
frank-large has no /metrics to scrape any more, and scraping it would be red by
construction: the halogen engine publishes /health, /cache and /v1/* only. That
job is removed with the reason written down (the remedy is the blackbox
exporter this fleet already wants for the comfyui/services probes), and
frank-small's llama.cpp endpoint takes the slot at 192.168.0.143:7771.

Homepage: frank-large's card still described the ik runtime, and the new
endpoint had no card at all. Both fixed, with a loopback probe -- the checker
runs on Dankclaw, which is also what makes the card green-able.

Dashboard: the endpoint picker was sourced from a vLLM-only metric, so a
llama.cpp job could never be selected, and the prose named frank-large as the
llama.cpp endpoint. The picker now matches both runtimes' series and the
llama.cpp targets carry the job filter every other panel already used.

The seams assert the new contract including the ABSENCE of a 7770 target, so a
well-meaning edit cannot re-add a permanently-red probe.
2026-09-16 13:59:49 -07:00
Frank 80a070d71b homepage: three cards could never be green, and one of them was lying
The board showed mariadb, valkey and Postgres as "down". None of them is down:
the checker runs on Dankclaw and can reach that host's loopback, its container
names, and published LAN addresses — while all three cards probe
container-internal database ports (192.168.0.235:3306, 127.0.0.1:6379/:5432).
They were red by construction, which is how a status board stops being read.
Publishing a database port on the LAN to make a badge green is not a trade this
fleet makes, so they are wip with the reason in their description, and the apps
that front them carry the status. Gluetun (WireGuard, UDP) was the pre-existing
example of the same call.

The rule is now written down in the Homepage doc: a card must be probeable from
the monitor's host or be wip. That is the same lesson as the Prometheus jobs
that scrape HTML pages and can never report up — an always-red signal teaches
the operator to ignore the surface it lives on.
2026-09-16 07:36:11 -07:00
Frank 56edfabb19 homepage: the Dankserver LLM card was labelled with a retired name
The card probing 192.168.0.235:7770 — frank-medium's endpoint since the
radiance cutover — was still called "frank-small", a service retired on
2026-09-15. The probe target was right; only the identity was wrong, which is
the worse half: a green card named after a service that no longer exists teaches
the wrong thing about the fleet. It is now frank-medium, described as what it
actually is (Qwen3.8-27B Uncensored MXFP4, vllm-radiance, two R9700s behind one
endpoint), and frank-large's description stops crediting the retired halogen
runtime for work ik_llama.cpp does.

The seam added to homepage_config.bats pins the cross-file contract that let
this drift: a card's probe target and its name must agree with the fleet, and a
retired service must not keep a card.
2026-09-16 06:53:20 -07:00
Frank af05c890fd observability: LLM alerting rules, evaluated but not yet delivered
The scraped metrics have no rules over them, so every graph was read after the
fact by a human. This adds shared/prometheus/rules/llm-fleet.yml, loaded via
rule_files and installed by the same copier that ships prometheus.yml (a rule
file nothing loads evaluates nothing, and Prometheus reports no error for it):

  LlmKvPoolSaturated      kv pool > 90% for 10m — the 122,636-token wall
  LlmRequestsQueueing     waiting > 0 for 10m — MAXSEQS=4 reached
  LlmPrefixCacheCollapse  reuse < 20% for 15m with traffic — the thrash signature
  LlmEndpointDown         scrape target down 2m — both replicas gone
  LlmLatencyHigh          p95 end-to-end > 60s for 10m — what an operator feels
  job:llm_prefix_cache_reuse:rate5m — recording rule shared by the alert and the
                          dashboard, so the two cannot drift

Every threshold is tied to something measured on this fleet, not to a guess,
and the file says plainly at the top that **nothing is notified**: there is no
Alertmanager here, so a firing alert reaches Prometheus' /alerts page, the
ALERTS metric and the dashboard panel added alongside it — and nowhere else.
Choosing how Frank actually gets paged is his call, not a missing config line.

Seam covers the silent failure mode: rules exist, the config references the
directory, the deployer installs it, and the dashboard graphs ALERTS.
2026-09-16 06:45:03 -07:00
Frank 6a191430ca observability: provision Grafana from git — datasource plus the LLM dashboard
Grafana has been running with zero datasources and zero dashboards, so the
metrics scraped last commit were only reachable through the Prometheus API.
This wires the plumbing and one dashboard, both declared in
shared/grafana/provisioning/ and installed to /opt/grafana/provisioning by the
deployer (the unit already mounted that directory read-only, so nothing about
the container changes):

- datasources/prometheus.yml — Prometheus at http://prometheus:9090, the
  container name, not the loopback publish, with a fixed uid so dashboards
  reference it without a lookup.
- dashboards/llm-fleet.json — KV pool fill, queue depth, prefix-cache reuse
  rate, e2e latency p95, and decode/prefill throughput, with an endpoint
  variable covering both frank-medium and frank-large. Every series name was
  checked against the live /metrics endpoint: this fleet already shipped one
  comment naming a metric that does not exist, and a dashboard full of those
  would be worse.

Dashboards stay editable in the UI (allowUiUpdates) because the panels are a
starting point, but the file is the source of truth and the deployer
overwrites Grafana's copy on every deploy.

Planner and seams follow the existing pattern: shared/grafana/** selects the
grafana service, and the tests pin the container-name address plus the
provisioning contract so a rename cannot silently empty the dashboard.
2026-09-16 06:39:15 -07:00
Frank 0f291115bd observability: name the prefix-cache series that actually exist
vllm:prefix_cache_hit_rate does not exist in this build; the exported pair is
vllm:prefix_cache_hits_total / vllm:prefix_cache_queries_total, whose rates give
the hit rate. A comment naming a series nobody can query is worse than no
comment.
2026-09-16 06:34:53 -07:00
Frank 42b47063ec observability: scrape the LLM fleet into Prometheus
Neither LLM endpoint was scraped, and frank-medium exports 418 vllm: series —
including the ones this fleet argues about under load: kv_cache_usage_perc (how
full the per-card pool is), prefix_cache_hit_rate (whether pinned conversations
are being reused), num_requests_waiting (queue depth) and the request-latency
histograms. Everything diagnosed tonight by hand — 122,636 KV tokens per card,
thrash when three large contexts overlap, cold-vs-warm prefix reuse — is a
graph now, on the LAN addresses rather than the guarded vhosts.

frank-large comes along in the same change: llama.cpp exposes throughput
counters under llamacpp:, and watching one LLM endpoint without the other was
arbitrary.

The seam added to observability_quadlet.bats pins the cross-file contract that
breaks silently: if a scrape target drifts from the :7770 the units publish,
the graphs vanish with no error anywhere.
2026-09-16 06:32:33 -07:00
Frank 6e6bdf7451 homepage: add Marreta as a user-category service
Adds the Marreta content-reader tile (http://marreta.franksdank.cloud,
probe 127.0.0.1:8082 via gluetun's publish) to the user grid so regular
authenticated users get a link to it. Registry-driven e2e picks it up
automatically.
2026-09-12 12:06:55 -07:00
Frank 301914b1c0 feat(llm): swap frank-large to halogen-flash-server, retire frank-npu
frank-large moves from the kingjones30/ROCmFPX llama.cpp build to
peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b
.hgn 115.55 GiB + quality overlay). Upstream measures this engine at
roughly 4x the prefill throughput of the llama.cpp family on the same
gfx1151 silicon, and prefill dominates any prompt with real context.

Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/),
engine on container loopback :8730 (never published), OpenAI front-end
:8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client
renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB,
conservative for a shared 121 GiB host), 4 slots, host reserve 24,
vision tower on, weights pinned, all other flags at vendor defaults.
Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable
/home/dank/models/halogen.

The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/
rather than deleted; the STRIX payload and the ROCmFPX build stay on
disk as the rollback target. deploy_llm_large becomes a user-quadlet
handler: it retires the old ROOT unit (moving its checkout symlink into
the home trash, never rm) and gates the first start on the download
rather than reporting a failure for a fetch that is working exactly as
designed.

frank-npu is retired in the same change. The kernel cmdline moves from
iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured,
and that takes the XDNA2 NPU with it: without PASID/SVA the driver
cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are
repointed (Hermes vision -> frank-large, Hermes title generation ->
frank-small, harness vision -> frank-large, tiny -> frank-small, task ->
frank-large:medium) and its provider blocks are removed from both the
harness config and Hermes's live config. Audio input leaves the fleet
with no replacement; image input survives on halogen's vision tower.

Docs: new halogen-flash-server runbook, new IOMMU decision record, and
frank-npu rewritten as a retirement record. AGENTS.md topology, runtime
sections, benchmarking policy (frank-large is a user unit now) and the
harness model-config section are corrected.
2026-09-11 14:48:28 -07:00
Frank 32b4016e55 feat(exposure): sonarr to admin side, internal-only by construction
Flip the homepage Sonarr card from the user category to admin and repoint
its link at the LAN-direct port (192.168.0.112:8989). Retire the unguarded
public vhost sonarr.franksdank.cloud (direct reverse_proxy, no auth) and its
post-auth @sonarr handler from the Caddyfile; drop the Pi-hole addn-hosts A
record with it. No public Porkbun record ever existed (NXDOMAIN at the zone),
so after this there is no WAN path to Sonarr: users request TV via Seerr
(guarded), admins reach Sonarr's own UI from the LAN only.

Fixes the seerr/sonarr mix-up from 19b2589/e0bade6.
2026-09-04 18:34:02 -07:00
Frank 4c7465d00d arr fleet: FrankBook-Pro is the sole deployment, all egress VPN-gated
FrankBook-Pro becomes the only *arr/qBittorrent deployment. Every fleet
container (qB + five arrs) now shares gluetun's netns, so all egress —
torrent, indexer APIs, metadata lookups — exits the WireGuard tunnel by
construction (AllowedIPs=0.0.0.0/0); the arrs' own ports publish on the
gluetun unit, none on the arr units themselves.

- dankserver/compose.yaml: remove the six arr services (the Aug 20 compose
  copy — never healthy, superseded; config dirs trashed on the host)
- dankclaw/Caddyfile + shared/homepage/services.json: arr vhosts and
  probes repoint 192.168.0.235 -> 192.168.0.112 (FrankBook-Pro)
- frankbook-pro units: five arrs join gluetun netns (Wants/After/Network);
  gluetun publishes 9696/8989/7878/8686/8787 alongside 8080/6881
- tests/frankbook_quadlet.bats: new VPN egress seam — every arr+qB unit
  must share gluetun's netns and publish no port; gluetun must publish the
  whole fleet's ports
- deploy-plan.py: gluetun changes carry the whole netns family (a gluetun
  recreation kills all six joiners); dankserver compose set loses the arrs
- AGENTS.md: deployment is complete only when unit tests AND the Playwright
  E2E suite both pass
2026-09-04 12:54:28 -07:00
Frank 19b2589362 Add arr stack to homepage admin section — sonarr on user side
Six arr services added:
- Sonarr → user section (after Seerr)
- Prowlarr, Radarr, Readarr, Lidarr, qBittorrent → admin section

All with hostname URLs (via Caddy vhosts) and probe ports on dankserver.
2026-09-04 01:03:36 -07:00
Frank 19675849aa Update service descriptions in shared/homepage/services.json 2026-09-03 22:54:09 -07:00
Frank 5797c0ee3f Format shared/homepage/services.json with proper indentation 2026-09-03 21:39:17 -07:00
Frank 8212d4451a docs: rename frank-default-large/small/sightsound to frank-large/small/npu across vault
Mechanical identity rename across the knowledge base, AGENTS.md, README, the
hermes config example, and homepage services.json. No behavior change; served
aliases on the LLM endpoints change with the deploy of commit 69f87c8.
2026-09-03 16:17:14 -07:00
Frank 834ad54112 homepage: probe Plex over TCP, not HTTP
PMS 1.43 serves TLS-only on 32400: a plain-HTTP GET gets a TLS alert and
curl reports "empty reply from server" (exit 52, code 000), so the card read
down while Plex answered 200 at /web/index.html over HTTPS. TCP connect is the
honest liveness signal for this port; the public URL stays HTTP because Caddy
terminates TLS and re-keys the path.
2026-09-02 13:51:25 -07:00
Frank 089255d45d homepage: repoint the Hermes probe at the live webui health port
The Hermes card probed 127.0.0.1:8090, a port retired when the hermes
stack moved from compose (8090 web dashboard) to quadlets: the dashboard
now publishes 9119 and the webui listens on 0.0.0.0:8787 in the host netns.
Nothing has listened on 8090 since, so the card reported "down" while the
gateway, dashboard, and webui were all running.

Probe 127.0.0.1:8787 instead — the webui runs HERMES_HOME=/data, the same
agent state as the gateway, and /health answers only when that server is
alive (uptime, sessions, active runs). The prober runs on Dankclaw, so the
loopback probe hits the host netns where the webui binds.
2026-09-02 13:40:44 -07:00
Frank 28f8dc2c0d homepage: point Plex link to /web/index.html 2026-09-01 20:45:49 -07:00
Frank ef1b077a2c homepage: move ComfyUI tile from user grid to admin panel
ComfyUI is internal-only since the 2026-08-31 vhost decommission; the
status tile belongs with the other admin-category services, not the
user grid.
2026-09-01 17:48:52 -07:00
Frank dfd9c8d1c0 e2e: Playwright suite for homepage + guard chain; RBAC gate on /admin; fix stale service links 2026-09-01 16:42:33 -07:00
Frank f4c9b4f381 seerr: quadlet on dankclaw + guard-chain vhost + deployer wiring
Unit: ghcr.io/seerr-team/seerr:v3.4.1, loopback-only 5055 publish
(homepage probe; users go through Anubis->Authentik->caddy:8017->seerr),
Containers data root, no env secrets. Guard chain: seerr.franksdank.cloud
vhost behind anubis:8923 with post-auth handler on 8017; addn-hosts record;
homepage widget; deploy-remote/deploy-plan/deploy.sh routing; seam test.
2026-09-01 14:40:10 -07:00
Frank 5829a0bfe8 docs: rename docs/ -> serverAppConfigs-docs/ per system-wide PROJECTNAME-docs convention
All in-repo path references (AGENTS.md, README, old scripts, report artifacts,
test code paths) rewritten in the same commit; focused-test map extended for
.gitignore, old/scripts/*, dankclaw/comfyui/*, and report/* per this repo's
own convention ('add one before committing'). Obsidian wikilinks are
title-based and unaffected.
2026-08-28 02:20:06 -07:00
Frank 2693259227 configure anonymous SearXNG provider defaults 2026-08-25 17:58:22 -07:00
Frank a05584b155 tune SearXNG VPN request timeouts and retries 2026-08-25 17:58:22 -07:00
Frank e7eaba4457 Homepage (TDD slice 7): fix the /admin/ path + status.json perms
Two real bugs found on the first production deploy:
- /admin/ 404'd: file_server looked for /srv/homepage/admin/ (a dir), but the
  page is admin.html. Add '@admin rewrite * /admin.html' so the /admin/ path
  serves admin.html.
- The pages used relative refs (style.css, app.js, services.json, status.json),
  which from /admin/ resolved to /admin/... (404). Add <base href="/"> so all
  refs resolve to the site root on both pages.
- status.json was written 0600 (root); the checker now chmods it 0644 so Caddy
  can serve it regardless of its user.
2026-08-23 16:49:33 -07:00
Frank 01c42202b8 Homepage (TDD slice 4): the pages, theme, and toggle
Red tests first, then the pages.
- tests/homepage_pages.bats: the pages reference the data + theme; no
  auto-refresh (Q41); no cursor alteration (Q28); the styling toggle strips
  to bare HTML (Q27); the footer shows the last master deployment datetime
  (Q43).
- index.html / admin.html: the user and admin pages. A grid of TUI panels,
  a summary line, the bare toggle, the deployed-at footer.
- app.js: reads status.json + services.json once (no auto-refresh), renders
  one panel per service for the page's category, the toggle, the footer.
- style.css: the terminal TUI theme (green-on-black, phosphor palette,
  scanlines) + a ?bare=1 mode that strips to unstyled HTML. No custom cursor.

Functionally verified with a Node DOM shim: 9 user panels, correct summary,
correct footer datetime.
2026-08-23 12:26:31 -07:00
Frank ce55d1dc3a Homepage (TDD slice 3): the real services.json + schema tests
Red tests first (the old services.json was an object, not the new list schema).
- tests/homepage_config.bats: the config is a JSON list; every entry has
  name/category/probe; category is user|admin; method is tcp|http; a
  showLink entry has a url and a status-only entry does not.
- shared/homepage/services.json: the real fleet. User: SearXNG, ComfyUI,
  OpenWebUI, Gitea, Calibre-Web, RomM, Copyparty, Plex, Jellyfin. Admin:
  Hermes, Grafana, Prometheus, Loki, node-exporter, alloy, Caddy, dnsmasq,
  Gluetun (wip - a tunnel has no simple local port), the two LLMs, mariadb,
  valkey, Postgres. Ports confirmed from the repo config (compose/quadlet).
  docs fields are empty for now (the /docs/ workstream fills them as each
  service goes active).
2026-08-23 12:11:21 -07:00
Frank 824d39a29e busybox-static for healthchecks; alloy: rely on the default UI listener
- alpine's /bin/busybox is musl-dynamic and cannot execute in the
  distroless loki image (missing loader); busybox-static links in
- alloy 1.x refuses to configure the ui service from a config block;
  the default UI already listens on 127.0.0.1:12345 where the
  healthcheck probes
2026-08-22 04:09:04 -07:00
Frank 1f4766f233 observability: fix alloy ui component, busybox healthchecks for loki/alloy
- alloy 1.x renamed the server component to ui; the old block failed the
  initial load and the container crash-looped
- grafana/loki is distroless and grafana/alloy lacks wget, so their
  healthchecks could never run; local images add a static busybox from
  alpine for that one command
- deploy.sh builds all three local images if missing
- tests: 3 new seams
2026-08-22 03:59:14 -07:00
Frank 765fd2902a searxng limiter: fix stale .local name in comment 2026-08-22 03:05:25 -07:00