Replaces llama.cpp Gemma4-12B-QAT (payload + binary retained as rollback).
SGLang in a self-contained python3.12 venv with an extracted CUDA 13.0
nvcc toolchain for the sgl-kernel/flashinfer JIT.
Shape: fp8 KV, pinned 606,208-token pool / 4 x 151,552 slots, bf16 mamba
states pinned at 32 slots (the engine autofills the ssm pool to a fixed
~5.1 GB regardless of dtype), eager prefill (no headroom for graph
capture), no spec decode (the checkpoint ships no mtp.* tensors despite
mtp_num_hidden_layers=1 — NEXTN drafts with an uninitialised 4.24 GiB
ghost at 0% acceptance).
Verified: 83.9 tok/s decode, 170k single prompt, 4 concurrent 54k
requests, vision (in-checkpoint tower), reasoning_content field.
The stock SearXNG image ships a native kagi engine; enabling it needs an
API key that must never enter git, and SearXNG does not expand env vars
in settings.yml (the live secret_key literally was '${SEARXNG_SECRET}'
until now — the app tolerated the placeholder string silently).
- settings.yml: kagi engine enabled (weight 2, kg shortcut, paging off so
exactly one API credit per aggregated query, timeout 8.0 vs 5.0 scrapers
so a slow paid query is not a wasted credit); api_key stays ${KAGI_API_KEY}.
- new scripts/searxng-render-settings.py: deploy-time renderer, allowlisted
to KAGI_API_KEY/SEARXNG_SECRET/SEARXNG_BASE_URL, fails the deploy loudly
on an unresolvable placeholder, idempotent on redeploys.
- deploy-remote.sh copy_searxng_config runs the renderer against
/opt/searxng/.env after copying settings.yml into /opt/searxng/config.
- KAGI_API_KEY placeholder added to dankclaw/.env.example (real key staged
into the host-side /opt/searxng/.env, mode 600, out of band).
- bats: engine wiring, renderer substitution, loud-failure, deployer seam.
- docs: SearXNG.md Kagi section + renderer law; Providers.md carve-out on
the no-credentials policy (paid engine acceptable when bounded + ordered).
Verified live on dankclaw: direct API probe via the gluetun netns (200,
data.search[] shape), then an aggregated query returned 29/29 results
tagged kagi, zero unresponsive engines.
The 2026-09-26 same-day cutover that ended the Lucebox DS4.1 detour:
Dankserver's :7770 endpoint is now unsloth llama.cpp (the b11030-mix
prebuilt, ROCm gfx120X, bundled runtime) serving
unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 — 175.30 GiB across six sha256-
verified split parts — on both Radeon AI PRO R9700s, as a ROOT unit (the
frank-large shape: no container, no generated unit).
The deployed contract:
- Placement law: layer split -ts 1,1 with -ngl 999; the routed experts
and the 51B-param n-gram (PLE) table are COMPUTED in host RAM via
-ot exps=CPU,ngram_embeddings=CPU (the x4-pair measurement: streaming
over PCIe loses at 3.7 t/s, computing in RAM does not care).
per_layer_token_embd.weight=CPU is required by the shared MTP head's
borrow path (a GPU-resident borrow crashes the draft load).
mmap stays on; --cache-ram 16384.
- Spec chain draft-mtp,ngram-mod: the shared Q8_0 MTP head
(mtp-...-shared-Q8_0.gguf, 2,786,568,256 B) at depth 2, with the
draftless ngram-mod matcher chained behind it — the only legal way to
layer two spec implementations in one process.
- Vision explicit: --mmproj names the BF16 projector (this build never
autoloads; a server without it comes up healthy and text-only).
- Reasoning: --reasoning-format deepseek → reasoning_content; the
Qwen3.8 template's {xhigh, medium, low} reasoning_effort gate folded
by reasoningEffortMap in models.yml.
- One stream, the model's full 262,144 context; greedy server default
with Qwen-recommended sampling riding compat.extraBody.
The deployer's deploy_llm_primary is rewritten for the root unit: it
gates every artifact at its declared byte count (stat -c %s, never wc -c
— which read the whole 377 GiB lucebox file for minutes on 2026-09-26),
retires the lucebox user quadlet to dated home trash, waits out port
7770, relinks/enables/restarts the root unit and waits up to 1800 s for
/health. The quadlet-era tree under dankserver/frank-primary/ is
archived (dankserver/.AGENTTRASH/20260926-llamacpp-cutover/ on this
box; git history everywhere else) and its planner mapping replaced by
the scripts/ trio.
Observability flips back to the llamacpp namespace: a frank-primary
Prometheus job at 192.168.0.235:7770 (cross-host LAN — the netavark
published-port quirk that forces host.containers.internal for the two
Dankclaw endpoints is same-host only), the LlmEndpointDown runbook
text, and the dashboard description for three llama.cpp endpoints.
Docs: the new runbook (frank-primary-llamacpp-qwen38-flash-next-q8.md),
RETIRED banners on the lucebox and radiance runbooks and their index
entries, the AGENTS.md fleet section rewritten for the root-unit shape,
and the homepage frank-medium card renamed to frank-primary with the
Q8_0 Flash-Next spec (docs.html #models and the modality note updated:
images on frank-primary and frank-small, audio on frank-small only).
models.yml: the dankserver provider is "llamacpp (dankserver)" serving
frank-primary, contextWindow 262144, maxTokens 65536 (compaction-gap
law: 183,501 + 65,536 <= 262,144), input [text, image].
Suite: 346 green (frank_primary_unit.bats rewritten for the root unit —
16 tests pinning the placement law, the shared-head requirement, the
explicit projector, HIP env hygiene (BIND_HOST, never bare HOST), and
the deployer's byte-count gates; the quadlet-era planner and
changed-tests mappings rewritten for the trio; the observability scrape
test asserts three llamacpp jobs again).
Rollback: the lucebox image (localhost/lucebox-rocm:gfx1201) and the
378 GiB payload stay on the host; the Qwen payload stays fetched, so a
re-swap in either direction is a redeploy, not a refetch.
The frank-medium retirement tail the 2026-09-26 lucebox cutover left
uncommitted: the radiance quadlet tree, its swap/fetch tooling and its
seam test are deleted (archive copies live in dankserver/.AGENTTRASH/
20260926-frank-medium-radiance/ and scripts/.AGENTTRASH/20260926-
frank-medium-radiance/), the planner/allowlist/test-changed mappings
drop the service, and frank-large's chat template moves to scripts/
(its old home was the retired frank-medium dir).
Finished here:
- tests/frank_primary_unit.bats: the lucebox quadlet seam (boot
persistence, host-network bind, HIP_VISIBLE_DEVICES=0,1, HIP-reserved
env-name guard, checkout-mounted entrypoint/healthcheck, first-start
patience, deployer payload/port gates).
- test-changed.sh: explicit .AGENTTRASH/ case — retirement commits move
files to trash, and the wrapper refused those additions with "no
focused-test mapping".
- observability: the vllm: namespace left with the radiance runtime.
Rules keep only llamacpp:* series (prefix-cache reuse, spec-decode,
queueing, endpoint-down); the dashboard drops the vLLM KV/latency
panels; prometheus.yml documents that frank-primary (lucebox) has no
/metrics and therefore no scrape job. llm-fleet.json was left
corrupted mid-edit (missing opening brace) and is rewritten from HEAD.
Suite: 339 green (was 7 red).
Dim the spec line (--dim) so the card reads pale role -> dim spec -> bright
spring-green inputs -> dim use. The inputs line now pops as the bright middle
tier and the spec/use are de-emphasized as reference detail.
Add --input: #00ff9d to the palette and point .desc-inputs at it, so the
Inputs line is visually distinct from the bright-green spec (--fg), the
pale-green role (--head), and the dim-green use (--dim).
Each LLM card now lists its accepted inputs as a line (Inputs: text · image ·
audio), sourced from the harness models.yml:
- frank-large: text (no vision projector)
- frank-medium: text, image (Gemma4 vision path)
- frank-small: text, image, audio (Gemma4 projector audio encoder; audio is
not in models.yml's input list only because OMP's schema rejects it, but the
endpoint accepts input_audio parts)
New .desc-inputs CSS + bare-mode reset; /docs model guide gains an Inputs line
per model. bats 30/30 + live render verified.
Each LLM card now carries a details object (role/spec/use) rendered as a
bold pale-green role, a bright-green spec line, and a dim 'Use:' line.
app.js descHTML() falls back to the plain description for non-LLM cards;
new .desc-role/.desc-spec/.desc-use CSS + bare-mode reset.
Roles made neutral (no nicknames): frank-large = deep reasoning,
frank-medium = primary model, frank-small = fast utility model.
The /docs model guide was fixed — it had drifted to retired model info
(frank-large as Qwen3.8-Flash-Next 87.3 GiB/200k, frank-medium as FP8
~31 GiB) — now Qwen3.8-27B UD-Q8_K_XL 31.5 GiB/262k and MXFP4/W4A8
19.4 GiB, reordered large->medium->small. bats 30/30 + live render.
frank-large: cybersecurity + debugging (requires Frank's permission)
frank-medium: primary intelligent model (surpasses Claude Sonnet 5 per
Artificial Analysis) — software engineering, sysadmin, instruction tasks
frank-small: fast/cheap utility model — agentic search, RAG, research
subagents, translation, roleplay, summarization, small utility tasks
Updated both the status cards (services.json) and the /docs model guide.
Footer link opens the authentik default-user-settings-flow in a new tab,
giving users a self-service entry point to change their password and
manage MFA/credentials without admin intervention.
The hermesadmin.franksdank.cloud dashboard (9119 — config, API keys,
sessions) is a distinct admin surface from the web UI client. Add it as a
third AI-section card; probe 9119 (302 login redirect counts as up, verified
on Dankclaw).
The admin Infrastructure section carried the retired 'dnsmasq' (probe-only,
no link) — Pi-hole FTL (a dnsmasq fork) replaced it on 2026-08-24. Replace it
with a linked Pi-hole card (pihole.franksdank.cloud, probe 8083 — the 403
basicauth counts as up). The Hermes card was probe-only; add the web UI
client link (hermes.franksdank.cloud). Both probes verified up on Dankclaw.
The user page now orders Web Apps, Media Library Apps, AI — the models sit
at the bottom, below the apps people click. The AI card and /docs#models
descriptions drop the infrastructure (runtimes, GPUs, tensor parallelism,
draft heads) in favor of what users care about: the hosted model, its
size, the context/parallel capacity, and what it's good at.
Each section's panels fill an auto-fill row (as many per row as the
viewport allows) and the sections stack vertically — the side-by-side
column layout was the wrong read of 'render the sections horizontally'.
The first cutover crash-looped the unit with 'invalid split file name':
the fetch script concatenated the three split parts into a single GGUF,
but part 1's header carries ggml.file_count=3, so the byte-correct
concatenation still advertises itself as a split and the loader demands
a -NNNNN-of-NNNNN filename.
- serve-frank-large.sh: -m points at part 1; llama.cpp loads the split
natively (parts 2-3 auto-discovered in the same directory)
- deploy-remote.sh: LLM_LARGE_MODEL gate path follows
- download-frank-large-unsloth.sh: assembly removed; the parts ARE the
canonical payload, verified against live Hub LFS oids
- docs: runbook model section + troubleshooting note, AGENTS.md, log
- the redundant 87 GiB assembled file is staged on dankclaw at
~/.AGENTTRASH/20260921-frank-large-assembled/
Both pages now group their panels under section headings rendered by shared
renderSection()/renderPanel() components in app.js (SECTION_ORDER fixes each
page's order: user = AI, Web Apps, Media Library Apps; admin = Media Library
Apps, Observability, AI, Infrastructure). The WIP badge splits into WIP: OPEN
and WIP: CLOSED via the wip value ("open"/"closed"); the checker passes the
value through as wip-open/wip-closed. The frank-large card follows the
unsloth llama.cpp cutover, frank-medium follows the FP8 checkpoint, and the
/docs model guide matches. Tautulli (Plex statistics, 192.168.0.235:8181)
joins the admin page's Media Library Apps section with an open link.
node-exporter is net=host, so its 192.168.0.143:9100 bind is a host bind,
not a published port — the bridge-networked Prometheus container cannot
reach its own host's LAN IP there (netavark DNATs published ports only).
The node-dankclaw target has been red by construction since the
observability cutover while the container reported healthy. Point it at
host.containers.internal:9100 (netavark delivers that to the host LAN IP,
where node-exporter listens) — the same rule frank-large and frank-small
already use. node-dankserver is a different machine and keeps its LAN
address. Update the cross-file contract test and the Targets table.
- shared/prometheus/rules/llm-fleet.yml: the llamacpp reuse recording rule
divided cached by prompt, but llama.cpp's prompt_tokens_total EXCLUDES
cached tokens (its own HELP line), so the ratio was unbounded — measured
77 on a fully-warm frank-large, meaning LlmPrefixCacheCollapse could never
fire on that side. Now cached / (cached + prompt), with the traffic floor
counting both terms too; the header documents the unit trap. Also fixed the
LlmEndpointDown description: frank-large's CIRU llama.cpp build DOES export
the llamacpp:* family (verified live), so the alert covers all three
endpoints.
- shared/grafana/provisioning/dashboards/llm-fleet.json: re-cut for two
namespaces. Every panel reads both (the endpoint picker selects); the
KV-pool-fill panel retired 2026-09-16 returned with
vllm:kv_cache_usage_perc; the latency p95 panel returned with the vLLM
histograms (TTFT/ITL/e2e); the alerts panel is scoped to the fleet's
Llm.* rules; the stale halogen-era description is gone. All 21 panel
expressions validated against the live endpoints.
- runbook + log + Observability: prefix caching measured, not inferred —
reuse happens at 1,648-token block granularity (attention block size padded
to the mamba page size); an identical 13,021-token prompt went 2.30 s cold
to 0.68 s warm with 9,888 tokens (76 %) reused, while a 2,521-token prompt
registered zero hits. Rule counts corrected to 6 alerts + 3 recording rules.
./tests/run.sh: 325 pass.
The radiance TP=2 cutover changed the metric namespace frank-medium exports
(llamacpp:* -> vllm:*) and left two scrape targets addressing host processes
by an address a container cannot reach.
- shared/prometheus/rules/llm-fleet.yml: two namespaces side by side. Added
LlmRequestsWaiting, LlmKvCacheSaturated (vllm-only gauges llama.cpp never
had), the job:llm_vllm_prefix_cache_reuse:rate5m and
job:llm_spec_accept_length:rate5m recording rules, and re-cut
LlmPrefixCacheCollapse / LlmSpecDecodeOff to OR both families with
per-family traffic floors so an idle endpoint cannot fire them. Every
expression validated against live Prometheus.
- shared/prometheus/prometheus.yml: frank-small -> host.containers.internal:7771
and frank-large scraped at host.containers.internal:7770. Both are HOST
processes on Dankclaw and netavark DNATs published container ports only, so
the LAN-IP targets were red by construction; frank-large's CIRU llama.cpp
build does export the llamacpp:* family, contra the old comment.
- tests/observability_quadlet.bats: the two seams now encode the new contract
(per-endpoint reachability by process type; every named series must be on the
measured list for its engine; engine-independent rules must name both).
- homepage docs.html/services.json: frank-medium sees pictures, frank-large is
CIRU Orca with a single seat (was halogen w4b, two seats, 200k).
- docs: radiance runbook is ACTIVE (deployed flags, three-arm battery, verified
capabilities, rollback, replica-era sections labelled); llama.cpp runbook
carries the SUPERSEDED banner; Dual-R9700-Cooperation states the measured
verdict; Observability records the namespace split and the DNAT lesson;
AGENTS.md topology/frank-medium/Model Config/Benchmarking Policy/runbook
index flipped; both indexes updated; work-log entry added.
Also repaired agent-pass damage: prose prepended above frontmatter openers,
duplicated runbook bodies, a literal tool-call token pasted into a runbook, and
an index destroyed by diff artifacts (restored from HEAD, flips re-applied).
Vault link check: 128 pages, 0 unresolved. ./tests/run.sh: 325 pass.
The conversation battery (scripts/bench-conversation.py, salted, same 10
prompts, 2026-09-20) measured three arms on the two R9700s:
arm I llama.cpp layer split 32:33 + DFlash2, Q8_0 geo decode 45.7 t/s, TTFT 0.72 s, burst-4 71.7 t/s
arm L llama.cpp tensor split 1:1 (RCCL), no drafter geo decode 30.5 t/s, TTFT 0.71 s, burst-4 54.3 t/s
arm R radiance vLLM TP=2 + DFlash2-FP8, MXFP4 geo decode 95.8 t/s, TTFT 0.21 s, burst-4 259.4 t/s
Radiance wins every axis, so it takes the alias. The board swap is what made
arm R possible: TP=2 was archived on 2026-09-16 because the second card sat
behind B650 chipset bridges with AtomicOpsCap: Routing-, and the ProArt
B850-Creator put both cards on CPU GPP root ports (Gen5 x8, Routing+, ReqEn+).
The R4D one-shot all-reduce needs no RCCL collective anyway.
- un-retire dankserver/frank-medium/: the TP=2 quadlet (banner rewritten —
it is the deployed shape now), radiance-entrypoint.sh, the healthcheck.
The replica pair and its affinity proxy stay archived; one process owns
both cards now.
- retire scripts/frank-medium.{service,env} + serve-frank-medium.sh and
bench-spec-ab.py to scripts/.AGENTTRASH/20260920-llama-layersplit/ as the
rollback path. build-llama-rdna4.sh and download-frank-medium-gguf.sh
stay: they are the tooling that rollback needs.
- deploy-remote.sh: deploy_llm_medium now gates on the image (pull once,
re-tag to the localhost name the quadlet resolves), the MXFP4 checkpoint,
the drafter, the pinned patch clone and the patched libr4d; retires the
llama root unit into the home trash instead of relinking it; then
install_quadlet + wait_container_healthy 1800s (first start compiles).
retire_vllm_medium_quadlets becomes retire_vllm_replica_quadlets — the
a/b/proxy trio only, since frank-medium is installed again.
- planner + test-changed map dankserver/frank-medium/** to frank-medium
and its seam; the scripts-trio mapping is gone with the files.
- tests/frank_medium_unit.bats rewritten for the container contract: boot
persistence of a GENERATED unit (no enable on this host's v1 generator),
TP=2 with the atomics-era NCCL workarounds kept out, the DFlash2 depth vs
the MXFP4 decode-kernel row band, the KV pin's coupling to the shape it
was calibrated at plus pool/VRAM arithmetic, the endpoint/tool/reasoning
contract, and the archive invariants.
- homepage card + docs updated, and they were stale twice over: they still
described the ajgazin Swift 35:30 / 393k payload, not the deployed shape.
Known regression, documented in the unit: vision. The MXFP4 checkpoint is
language_model_only and the launcher loads no projector, so images are a 400
here; the harness's vision role already routes to frank-small.
- services.json: frank-large/medium/small move from admin-only to the user
page with current descriptions (halogen w4b iGPU; 35:30 layer-split MTP
27B on the R9700s; Gemma4-12B QAT vision path on the 3090) and docs
links to /docs#models
- docs.html: static user documentation (access chain, model guide, media
how-tos, troubleshooting); served at /docs via the post-auth homepage
listener; shipped by deploy_homepage; footer + index link to it
- tests: pages seam asserts the docs contract (page, link, Caddy route,
deployer copy)
- vault: Homepage.md synced (docs landing shipped, cards reclassified)
169.254.1.2 is not a local address on this host (netavark runs its
bridges in aardvark-dns' netns), so the relay cannot bind it — and a
loopback bind refuses container traffic. Probe-socket proof: a host
socket on 192.168.0.143:18094 answers the container's 169.254.1.2:18094
dial while the LAN IP refuses from the container. The relay binds
192.168.0.143:18093 and the deployer installs an iptables INPUT pair
(ACCEPT 10.89.0.0/16, DROP rest) so the unauthenticated endpoint stays
off the LAN.
A socket bound to 127.0.0.1 refuses container traffic that arrives via
the podman bridge: the first live webhook failed with 'dial tcp
169.254.1.2:18093: connection refused'. The relay now binds the
host-gateway endpoint — link-local to the podman bridge, off the LAN —
and the deployer's health probe follows it there.
Context: 262144 was measured before it was deployed -- 24.72 GB on GPU 0 and
29.14 GB on GPU 1 of 34.21 GB each, ~5 GB of headroom on the busier card, and a
140,069-token prompt prefilled through the single slot in 134 s to prove the
window is real. The 114688 wall was the vLLM predecessor's arithmetic, not this
runtime's.
Throughput, measured through the service: 1.0-1.4k tok/s prefill from 4k to
140k, 43.8 tok/s decode on a 512-token generation, 41-71 % DFlash2 acceptance,
and the prefix cache re-prefilling 4 of 998 tokens on a repeated prompt.
The signals move with the runtime, so the fleet's rules and dashboard are
re-cut rather than renamed: vllm: series do not exist any more, and two of the
old rules watched things llama.cpp does not export at all (KV-pool fill, p95
latency). Those are retired with the reason written down, and LlmSpecDecodeOff
takes one of the slots -- a lost drafter is the failure no client can see. The
observability seam now fails if any rule names a series the endpoint does not
export, which is what would have caught the stale vllm: expressions.
omp models.yml (outside the repo) is updated in the same pass: contextWindow
262144, and the full reasoning-effort map, because Qwen3.8's template answers
500 on 'high' -- verified against the live endpoint, low/medium/xhigh answer
200. Subagents run at high.
The runbook for the new deployment, plus the sweep of everything that still
described the vllm-radiance replica pair: the vault index and its ACTIVE
marker, the superseded banner on the radiance page, the pointer from the
ROCmFPX page, the homepage card, the Grafana dashboard description and the
observability page's scrape table.
The new page carries the two facts a future agent needs and the flags do not
say: llama.cpp's layer split is what lets BOTH cards serve ONE model without
any collective (vLLM's PP handoff is an RCCL send/recv, which is the primitive
this host cannot dispatch), and the rocBLAS Tensile-path trap that makes a
healthy-looking build fail on the first request.
frank-large has no /metrics to scrape any more, and scraping it would be red by
construction: the halogen engine publishes /health, /cache and /v1/* only. That
job is removed with the reason written down (the remedy is the blackbox
exporter this fleet already wants for the comfyui/services probes), and
frank-small's llama.cpp endpoint takes the slot at 192.168.0.143:7771.
Homepage: frank-large's card still described the ik runtime, and the new
endpoint had no card at all. Both fixed, with a loopback probe -- the checker
runs on Dankclaw, which is also what makes the card green-able.
Dashboard: the endpoint picker was sourced from a vLLM-only metric, so a
llama.cpp job could never be selected, and the prose named frank-large as the
llama.cpp endpoint. The picker now matches both runtimes' series and the
llama.cpp targets carry the job filter every other panel already used.
The seams assert the new contract including the ABSENCE of a 7770 target, so a
well-meaning edit cannot re-add a permanently-red probe.
The board showed mariadb, valkey and Postgres as "down". None of them is down:
the checker runs on Dankclaw and can reach that host's loopback, its container
names, and published LAN addresses — while all three cards probe
container-internal database ports (192.168.0.235:3306, 127.0.0.1:6379/:5432).
They were red by construction, which is how a status board stops being read.
Publishing a database port on the LAN to make a badge green is not a trade this
fleet makes, so they are wip with the reason in their description, and the apps
that front them carry the status. Gluetun (WireGuard, UDP) was the pre-existing
example of the same call.
The rule is now written down in the Homepage doc: a card must be probeable from
the monitor's host or be wip. That is the same lesson as the Prometheus jobs
that scrape HTML pages and can never report up — an always-red signal teaches
the operator to ignore the surface it lives on.
The card probing 192.168.0.235:7770 — frank-medium's endpoint since the
radiance cutover — was still called "frank-small", a service retired on
2026-09-15. The probe target was right; only the identity was wrong, which is
the worse half: a green card named after a service that no longer exists teaches
the wrong thing about the fleet. It is now frank-medium, described as what it
actually is (Qwen3.8-27B Uncensored MXFP4, vllm-radiance, two R9700s behind one
endpoint), and frank-large's description stops crediting the retired halogen
runtime for work ik_llama.cpp does.
The seam added to homepage_config.bats pins the cross-file contract that let
this drift: a card's probe target and its name must agree with the fleet, and a
retired service must not keep a card.
The scraped metrics have no rules over them, so every graph was read after the
fact by a human. This adds shared/prometheus/rules/llm-fleet.yml, loaded via
rule_files and installed by the same copier that ships prometheus.yml (a rule
file nothing loads evaluates nothing, and Prometheus reports no error for it):
LlmKvPoolSaturated kv pool > 90% for 10m — the 122,636-token wall
LlmRequestsQueueing waiting > 0 for 10m — MAXSEQS=4 reached
LlmPrefixCacheCollapse reuse < 20% for 15m with traffic — the thrash signature
LlmEndpointDown scrape target down 2m — both replicas gone
LlmLatencyHigh p95 end-to-end > 60s for 10m — what an operator feels
job:llm_prefix_cache_reuse:rate5m — recording rule shared by the alert and the
dashboard, so the two cannot drift
Every threshold is tied to something measured on this fleet, not to a guess,
and the file says plainly at the top that **nothing is notified**: there is no
Alertmanager here, so a firing alert reaches Prometheus' /alerts page, the
ALERTS metric and the dashboard panel added alongside it — and nowhere else.
Choosing how Frank actually gets paged is his call, not a missing config line.
Seam covers the silent failure mode: rules exist, the config references the
directory, the deployer installs it, and the dashboard graphs ALERTS.
Grafana has been running with zero datasources and zero dashboards, so the
metrics scraped last commit were only reachable through the Prometheus API.
This wires the plumbing and one dashboard, both declared in
shared/grafana/provisioning/ and installed to /opt/grafana/provisioning by the
deployer (the unit already mounted that directory read-only, so nothing about
the container changes):
- datasources/prometheus.yml — Prometheus at http://prometheus:9090, the
container name, not the loopback publish, with a fixed uid so dashboards
reference it without a lookup.
- dashboards/llm-fleet.json — KV pool fill, queue depth, prefix-cache reuse
rate, e2e latency p95, and decode/prefill throughput, with an endpoint
variable covering both frank-medium and frank-large. Every series name was
checked against the live /metrics endpoint: this fleet already shipped one
comment naming a metric that does not exist, and a dashboard full of those
would be worse.
Dashboards stay editable in the UI (allowUiUpdates) because the panels are a
starting point, but the file is the source of truth and the deployer
overwrites Grafana's copy on every deploy.
Planner and seams follow the existing pattern: shared/grafana/** selects the
grafana service, and the tests pin the container-name address plus the
provisioning contract so a rename cannot silently empty the dashboard.
vllm:prefix_cache_hit_rate does not exist in this build; the exported pair is
vllm:prefix_cache_hits_total / vllm:prefix_cache_queries_total, whose rates give
the hit rate. A comment naming a series nobody can query is worse than no
comment.
Neither LLM endpoint was scraped, and frank-medium exports 418 vllm: series —
including the ones this fleet argues about under load: kv_cache_usage_perc (how
full the per-card pool is), prefix_cache_hit_rate (whether pinned conversations
are being reused), num_requests_waiting (queue depth) and the request-latency
histograms. Everything diagnosed tonight by hand — 122,636 KV tokens per card,
thrash when three large contexts overlap, cold-vs-warm prefix reuse — is a
graph now, on the LAN addresses rather than the guarded vhosts.
frank-large comes along in the same change: llama.cpp exposes throughput
counters under llamacpp:, and watching one LLM endpoint without the other was
arbitrary.
The seam added to observability_quadlet.bats pins the cross-file contract that
breaks silently: if a scrape target drifts from the :7770 the units publish,
the graphs vanish with no error anywhere.
Adds the Marreta content-reader tile (http://marreta.franksdank.cloud,
probe 127.0.0.1:8082 via gluetun's publish) to the user grid so regular
authenticated users get a link to it. Registry-driven e2e picks it up
automatically.
frank-large moves from the kingjones30/ROCmFPX llama.cpp build to
peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b
.hgn 115.55 GiB + quality overlay). Upstream measures this engine at
roughly 4x the prefill throughput of the llama.cpp family on the same
gfx1151 silicon, and prefill dominates any prompt with real context.
Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/),
engine on container loopback :8730 (never published), OpenAI front-end
:8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client
renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB,
conservative for a shared 121 GiB host), 4 slots, host reserve 24,
vision tower on, weights pinned, all other flags at vendor defaults.
Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable
/home/dank/models/halogen.
The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/
rather than deleted; the STRIX payload and the ROCmFPX build stay on
disk as the rollback target. deploy_llm_large becomes a user-quadlet
handler: it retires the old ROOT unit (moving its checkout symlink into
the home trash, never rm) and gates the first start on the download
rather than reporting a failure for a fetch that is working exactly as
designed.
frank-npu is retired in the same change. The kernel cmdline moves from
iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured,
and that takes the XDNA2 NPU with it: without PASID/SVA the driver
cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are
repointed (Hermes vision -> frank-large, Hermes title generation ->
frank-small, harness vision -> frank-large, tiny -> frank-small, task ->
frank-large:medium) and its provider blocks are removed from both the
harness config and Hermes's live config. Audio input leaves the fleet
with no replacement; image input survives on halogen's vision tower.
Docs: new halogen-flash-server runbook, new IOMMU decision record, and
frank-npu rewritten as a retirement record. AGENTS.md topology, runtime
sections, benchmarking policy (frank-large is a user unit now) and the
harness model-config section are corrected.
Flip the homepage Sonarr card from the user category to admin and repoint
its link at the LAN-direct port (192.168.0.112:8989). Retire the unguarded
public vhost sonarr.franksdank.cloud (direct reverse_proxy, no auth) and its
post-auth @sonarr handler from the Caddyfile; drop the Pi-hole addn-hosts A
record with it. No public Porkbun record ever existed (NXDOMAIN at the zone),
so after this there is no WAN path to Sonarr: users request TV via Seerr
(guarded), admins reach Sonarr's own UI from the LAN only.
Fixes the seerr/sonarr mix-up from 19b2589/e0bade6.
FrankBook-Pro becomes the only *arr/qBittorrent deployment. Every fleet
container (qB + five arrs) now shares gluetun's netns, so all egress —
torrent, indexer APIs, metadata lookups — exits the WireGuard tunnel by
construction (AllowedIPs=0.0.0.0/0); the arrs' own ports publish on the
gluetun unit, none on the arr units themselves.
- dankserver/compose.yaml: remove the six arr services (the Aug 20 compose
copy — never healthy, superseded; config dirs trashed on the host)
- dankclaw/Caddyfile + shared/homepage/services.json: arr vhosts and
probes repoint 192.168.0.235 -> 192.168.0.112 (FrankBook-Pro)
- frankbook-pro units: five arrs join gluetun netns (Wants/After/Network);
gluetun publishes 9696/8989/7878/8686/8787 alongside 8080/6881
- tests/frankbook_quadlet.bats: new VPN egress seam — every arr+qB unit
must share gluetun's netns and publish no port; gluetun must publish the
whole fleet's ports
- deploy-plan.py: gluetun changes carry the whole netns family (a gluetun
recreation kills all six joiners); dankserver compose set loses the arrs
- AGENTS.md: deployment is complete only when unit tests AND the Playwright
E2E suite both pass
Mechanical identity rename across the knowledge base, AGENTS.md, README, the
hermes config example, and homepage services.json. No behavior change; served
aliases on the LLM endpoints change with the deploy of commit 69f87c8.
PMS 1.43 serves TLS-only on 32400: a plain-HTTP GET gets a TLS alert and
curl reports "empty reply from server" (exit 52, code 000), so the card read
down while Plex answered 200 at /web/index.html over HTTPS. TCP connect is the
honest liveness signal for this port; the public URL stays HTTP because Caddy
terminates TLS and re-keys the path.
The Hermes card probed 127.0.0.1:8090, a port retired when the hermes
stack moved from compose (8090 web dashboard) to quadlets: the dashboard
now publishes 9119 and the webui listens on 0.0.0.0:8787 in the host netns.
Nothing has listened on 8090 since, so the card reported "down" while the
gateway, dashboard, and webui were all running.
Probe 127.0.0.1:8787 instead — the webui runs HERMES_HOME=/data, the same
agent state as the gateway, and /health answers only when that server is
alive (uptime, sessions, active runs). The prober runs on Dankclaw, so the
loopback probe hits the host netns where the webui binds.
Frank
