Loki metrics queries group only by indexed stream labels, so the
journal-as-JSON stream made every per-container view structurally
empty (measured 6.5k lines/5min flowing while the board read zero).
A loki.process relabel stage now extracts CONTAINER_NAME into a
'container' label. alloy 1.21's json stage has no on_error attribute
(validated against the pinned image) — format_as_json guarantees the
line shape, so no error branch exists.
observability.json: stale 'structural downs' panel description fixed,
target census matches the post-blackbox job set. logs-overview.json:
panels grouped by the promoted container label instead of the absent
container_name, so the board stops reading empty beside live traffic.
Prometheus 3 refuses HTML scrape bodies, so the comfyui/services probe
jobs were structurally dead targets reading as outages. HTTP liveness
now rides a blackbox exporter (shared/blackbox/blackbox.yml, loopback
bind, http_2xx module); prometheus.yml probes SearXNG, OpenWebUI and
ComfyUI via __param_target relabeling. The unit is a boot-persistent,
self-healing root quadlet.
Census: all 5 provisioned dashboards, 56 targets, every expression
re-evaluated against the live Prometheus/Loki with template vars expanded
(queries must run from the host: both APIs bind 127.0.0.1; from the dev box
that is connection-refused, which is NOT a dashboard defect).
- host-system 'Load 5m / core' was STRUCTURALLY EMPTY: bare '/' drops
non-matching pairs and the right side collapses to {job} while the left
carries {job,instance} -> 0 series forever. Fixed with / on(job); proven
live (2 series).
- containers 'Memory vs limit (%)' was STRUCTURALLY EMPTY: every
container_spec_memory_limit_bytes on this fleet is 0 (no cgroup limits),
so the '> 0' filter can never match. Rewritten to share-of-host-total via
label_replace join of the node exporter's MemTotal by host (the parens
form is a PromQL parse error: nothing may follow group_left); 62 series
live. Panel unit percentunit kept -> expression returns a fraction.
- llm-fleet 'Firing fleet alerts' is legitimately empty when healthy
(ALERTS carries only firing/pending series); description now says so,
per the Stale Signal rule that 0-series-on-ALERTS is not a defect.
logs-overview and observability: all targets already returning data.
Two shipped-2026-10-20 bugs that left gpu-dankclaw, gpu-dankserver and
cadvisor-dankclaw DOWN beside a green deploy:
- gpu-exporter.py built one ThreadingHTTPServer per bind but ran
serve_forever() on servers[0] only: the LAN-class socket sat listen()ing
but never accept()ed, loopback deploy gate passed, every scrape that
reached the other bind timed out on the SYN backlog. Now one thread per
extra bind. Regression test spins --serve on two loopback binds and
requires BOTH to answer /metrics.
- cadvisor-dankclaw scraped its own host's LAN IP. Measured from inside the
Prometheus container: a bridge container cannot dial its own host's LAN
address at all (hairpin refuses published ports too); the podman gateway
answers. Target moved to host.containers.internal:8080; the seam that
pinned the LAN address (asserting the wrong doctrine) is re-pinned to the
gateway plus a negative pin against the own-host LAN dial. Observability
page records the measured reachability law replacing the inferred one.
Add a host-process GPU exporter, scrape it on both servers, and graph it
on the host-system dashboard. The fleet has four GPUs across three vendors
of interest (2x R9700 on Dankserver, the 8060S iGPU + RTX 3090 on Dankclaw)
and nothing exposed them until now.
scripts/gpu-exporter.py reads each AMD card's amdgpu sysfs
(gpu_busy_percent / mem_info_vram_* / hwmon temp1_input / power1_average)
plus one nvidia-smi --query-gpu CSV, and emits unified gpu_* Prometheus
families (utilization_percent, memory_used/total_bytes, temperature_celsius,
power_watts) labelled gpu/vendor/model. Absent hardware yields ABSENT series,
never a fabricated 0: the R9700s expose no sysfs power, so they simply carry
no gpu_power_watts line. Both sources are injectable, so the seam runs on a
GPU-less dev box; live-verified on Dankserver's real sysfs.
It runs as a plain user unit (not a quadlet): it must read the host's own DRM
sysfs and shell to nvidia-smi, which a container cannot do without GPU
passthrough. Same posture as signal-relay. The base unit binds loopback; the
deploy drop-in appends the host LAN address, never a wildcard (matching
node-exporter/cadvisor). deploy_gpu_exporter installs/enables/health-gates it
on both hosts and is wired into the planner (both targets), the test-changed
router, and two Prometheus scrape jobs using the node/cadvisor reach rules
(gateway on Dankclaw, LAN on Dankserver).
Per-container CPU/memory was invisible for the quadlet app stack — Grafana could
only see the host (node_exporter) and the LLM engines. This adds a rootless
cadvisor quadlet on each host and a `containers` dashboard.
cadvisor works on this rootless-Podman / cgroup-v2 host on one condition:
--cgroupns=host. Podman starts every container in a private cgroup namespace, so
without it cadvisor sees only its own cgroup root and exports a single series
(measured 2026-10-20: 1 series off, 996 on). The libpod socket does not name the
rootless app containers (it resolved 2 of 34), so naming comes from elsewhere:
the quadlet unit name is carried in the cgroup path
(app.slice/<name>.service/libpod-payload-<hex>), and Prometheus relabels it into
a `container` label. No podman socket needed.
cadvisor cannot bind per-interface like node_exporter, so each unit is a bridge
container publishing exactly 127.0.0.1:8080 and the host LAN address — never a
wildcard. Prometheus (a bridge container) reaches a host-published port at the
LAN address, so cadvisor-dankclaw/dankserver scrape 192.168.0.143:8080 /
192.168.0.235:8080. Only the families cadvisor actually exports for these
containers are graphed — CPU usage, memory working-set/limit, OOM — not network
or filesystem, which are absent under the host-PID layout.
Wired through the deployer (allowlist, deploy_observability case + /healthz gate,
dispatcher), the planner (both targets), and observability_quadlet.bats.
llm-fleet.json already owns LLM capacity+spec-decode. These three cover the
domains that had NO dashboard:
- host-system: node_exporter CPU/Load/Mem/Disk/Net/Uptime/FS/FDs for both
Dankclaw and Dankserver (templated by job node-dankclaw|node-dankserver)
- logs-overview: Loki volume + per-container error rate + a sample log panel;
container is parsed from the JSON line at query time (journal streams have
no container_name stream label)
- observability: the stack's own health - Prometheus target up/down + scrape
interval drift + samples/s + TSDB, Loki ingest/discard, Grafana CPU/mem.
Panel queries verified live against Prometheus/Loki; provisioned dashboards
confirmed via /api/search. Alertmanager is running but NOT scraped by
Prometheus - noted in the dashboard, no empty panels built on it.
A rootless Alloy shipped zero lines: the journal files are
root:systemd-journal 0640 (other::---) and a rootless container carries no
supplementary groups, so sd_journal_open sees an empty view silently; the
rootless quadlet container logs also live in user-1000.journal, not the
ctr.log paths the old config tailed.
As root, one loki.source.journal reads system.journal + user-*.journal
and carries both. Network=host because Loki publishes 127.0.0.1:3100 and
no bridge container can reach the host loopback; the UI binds loopback
directly (no PublishPort). The /etc/machine-id mount is load-bearing —
sd_journal finds the journal dir by the local machine id; without it the
source reads 0 lines, health green.
deploy_alloy routes around the rootless loop: build rootless, mirror the
image into the root store (a root build fails on DNS in the build
container), retire the old rootless alloy unit, install to
/etc/containers/systemd. Validated with a throwaway root probe: 31,825
lines read, 26,966 pushed to Loki in 20s, zero drops.
Adds shared/grafana/provisioning/datasources/loki.yml at http://loki:3100
so logs are readable through Grafana Explore. Loki ships no web UI of its
own and the guarded vhost serves only the API; this is the human surface.
radiance exports vllm:* metrics (verified on the live endpoint), not
llamacpp:*: nothing in llm-fleet could ever see frank-archon. Add a
vllm-fleet group - 1 recording + 4 alert (queueing, prefix-cache
collapse, preemption, p95 time-to-first-token) keyed to vllm:*.
Split the observability seam so llm-fleet rules bind llamacpp:* and
vllm-fleet rules bind vllm:*, and neither may evaluate the other's
namespace. Drops the retired no-metrics-for-archon doctrine the seam
still asserted and the vacuous namespace grep.
open-notebook has no user accounts upstream (PasswordAuthMiddleware = one
shared password; SurrealDB has no user/account/owner table — verified in the
running image). One shared password = one shared workspace; the guard chain
gates access but cannot partition data, and Authentik identity is never
consumed. Multi-user is approved-but-unbuilt upstream (#712, post-v2); OIDC
declined (#607). So move the homepage card from the user grid to the admin
tier and add the single-tenant boundary to the vault page. Fix the card's
stale 'frank-zealot' -> frank-archon. Update the seam's incidental category
pin to the new intentional value. Password rotated on the host (not repo).
The service came up internal-only: UI on the LAN at 8502, API at 5056, both
published on every interface, with only its own shared password in front. That
put the episode library outside the fleet's auth chain and one port scan away
from anything on the network.
- Caddy: guarded vhost notebook.franksdank.cloud (Anubis -> Authentik; provider
notebook-proxy, internal_host caddy:8017) plus the post-auth route to the
published UI port. One upstream is enough, because the image's own Next.js
layer rewrites /api/* to the FastAPI beside it — so the API needs no outward
face at all, and 5056 moves to loopback, where only the deployer's health gate
dials it.
- scripts/open-notebook.env: API_URL and CORS_ORIGINS become the single guarded
origin. Left to itself the app auto-detects <host>:5055, the port seerr holds
on this host; a LAN-origin API_URL would also take the browser around the
guard chain, where its authenticated session does not ride.
- addn-hosts: the LAN A record. The homepage card goes live and links the vhost.
- Seam: the API origin is derived from whatever the Caddyfile routes to 8502, so
renaming the vhost moves the test instead of leaving it blessing a stale name,
and a new test proves the vhost hands the WHOLE request to Anubis.
The endpoint deployed on 2026-10-18, so the admin card must carry its state rather than a placeholder: wip becomes false (the registry schema wants the key on every service; false is what the live cards use) and the description drops the "Hidden until deployed" line. showLink stays false and the card stays admin-only: an HTTP API has no page to open, and homepage-status.py counts ANY HTTP response as up, so the card is honest with no link and no /health path in the probe. The voice seam that pinned wip == down is re-keyed to the deployed contract and now also refuses a link or a user-category placement.
Stage 2 of the voice stack. Repo state only: nothing deployed, no host touched,
no container started.
- `frank-tts` is now registered everywhere a dankclaw service has to be:
`SERVICE_ORDER` (after `frank-tts-engine`, before the consumers), the planner's
dankclaw set, `deploy-remote.sh`'s allowlist + dispatch + a
`deploy_frank_tts` handler, and `deploy.sh`'s dispatch regex — plus the
`build_compose_image` arm naming `dankclaw/voiceservice/Containerfile`, without
which the build silently returns 0 and the unit's `Pull=never` tag never
exists.
- The unit joins TWO networks, in order: `dankclaw_default` first (default route,
where consumers and open-notebook live), then the voice bridge. `TTS_UPSTREAM`
moves from `host.containers.internal:8880` to `http://frank-tts-engine:8880` —
the actual fix for a shape that could never work, since a bridge container
cannot reach a host-loopback bind and the engine publishes nothing at all.
- `Wants=`/`After=` on the engine, never `Requires=`: the `llama-tts-cli` rollback
backend needs no upstream, so a broken engine may delay this endpoint but must
not take its unit down.
- The handler installs the engine's quadlet pair too, because a referenced
`Network=` file the generator cannot find means no façade unit is produced at
all, and `--only frank-tts` has no other way to get it. It creates
`/home/dank/voices` as the deploy user (rootless podman would auto-create a
missing bind source root-owned, which reads as "no voices"), and gates on
podman's recorded health.
- Voice naming is now written down where it can be found: a voice is a lowercase
directory name matching `^[a-z0-9][a-z0-9._-]*$`, at most 64 chars. That shape
is the intersection of two rules that disagree — the shim would accept mixed
case, the engine lowercases names — so a mixed-case pack resolves on one side
only, and the boot warmer refuses it loudly. `narrator1/2/3` are load-bearing
for open-notebook.
- Homepage card for the pair, `wip: down` and hidden, probing `127.0.0.1:7772`;
the engine gets no card because it publishes nothing and could never be probed.
- **No Prometheus job, deliberately.** No `/metrics` exists on the shim, this
Prometheus 3 refuses HTML/JSON bodies (the existing `comfyui` and `services`
probes are documented down for exactly that), and `LlmEndpointDown` is scoped
`up{job=~"frank-.*"}` — so a `/health` job would page on Signal for a healthy
endpoint. Recorded in the page instead of quietly skipped.
- Vault: new [[Frank TTS]] page linked from the index; [[Frank TTS Engine]] now
points at it for the pair's public signal; log entry for the day.
- Seam grows to 25 tests: the generator's rendering of the pair (both
attachments, first-network order, one publish, Wants-not-Requires, bounded
per-unit blocks), the env file's upstream checked as a network JOIN rather than
a string, planner separation and order, the handler's steps, the card's gating,
routing that selects both seams for either deployable.
Internal-only: port 8502 direct, no vhost field, no Caddy change — exposure
stays behind an explicit decision, and the app carries its own password.
urlStatus=hidden is deliberate: the homepage probe runs on the gateway and the
pair is not deployed, so a live card would render dead on day one. Flipping it
to live is the only homepage change the deploy needs; the seam asserts the
hidden state while the service is repo-only and requires the required card
fields either way.
mono-on.svg follows the mono-* icon set the sibling cards use, so the card does
not fall back to the generic dot.
Swap the 3090 endpoint from Ternary-Bonsai-2-27B PQ2_0 (2 x 200,704,
fork build 10754) to Ternary-Bonsai-2-9B PQ2_0 with the MTP head IN the
file, three conversations at the trained 262,144 window
(--ctx-size 786432 --parallel 3), on vendor fork build 10777 -- the
minimum build that pairs MTP with a quantized V cache. q8_0 K + q8_0 V
is the only KV that fits: 13,056 MiB against a 26,112 MiB f16 pair.
Settled boot 17,876 used / 6,249 free.
MTP is armed and then DECLINED by measurement: the draft context bills
its own f16 KV over the whole pool (3,072 MiB) and at UBATCH_SIZE=2048
that halves decode (167.2 -> 109.2 t/s). The documented MTP arm
(SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024, 213.5 t/s) is recorded in the
env header rather than left to rediscovery.
Two bugs the trial caught: ${SPEC_PRESENCE_PENALTY:=0.0} was a
default-assignment, so the env value always won and the no-presence-penalty
MTP arm was unreachable (now ${VAR=0.0}); and the Q8_0 mmproj is the
publisher's requirement for concurrent image requests, which matters at
--parallel 3.
contextWindow is now 262144 -- the widest in the fleet -- so :7771 is the
compaction hub. 27B runbook re-banned RETIRED with its bench-window
section closed; indexes, homepage card, OpenWebUI doc and the worklog
re-keyed.
- card/docs: prism-ml payload link, PQ2_0 spec, ~77 t/s (measured single
stream, replacing the 9B's ~95)
- warning re-keyed: zealot is no longer an uncensored endpoint, so the card
carries the base-alignment notice and points uncensored work at dragoon;
the test's "common uncensored text" equality (zealot==dragoon) is retired
with the payload it described — dragoon alone carries the legal warning
Model swap on the radiance endpoint: MODEL_FILE moves from the in-house
qwen4exp-swift15-w4dn8-i8-hc8m.rad (116,416 MiB) to the engine author's
published qwen3.8-next-flash-fp8-iq4r-moe.rad (116,322 MiB, official
Qwen3.8-Flash-Next bf16 -> 4-bit non-uniform codebook experts + int8
trunk, card KL 0.0789 / top-1 92.85 % vs bf16). NO argument changed:
25600/5120 vram pair, 44928 host pool, fp8 KV, 3 x 200000, MTP depth 2,
wht6, MNBT 2048, froggeric v22.5 — the 2026-10-13 shape carries verbatim.
New fetch script scripts/download-frank-archon-iq4r.sh (hfget + exact
byte count 121,969,901,568 + LFS sha256, --verify-only default, ungated,
no token on the command line). Unit/serve/deployer lore re-keyed; the
seam pins the new payload id + sha256 and keeps the w4dn8 rollback
provenance. Homepage card points at the publisher repo.
Rollback: revert this commit (w4dn8 .rad stays on disk). The fp8-native
rad was deleted from dankserver 2026-10-15 (Frank's one-time rm, volume
for this payload); rebuild path documented.
The 2026-10-09 wire aliases become the service names everywhere:
systemd units, serve scripts, env files and FRANK_* variables,
deploy-plan/deploy-remote service keys, container names, Prometheus
job names + alert rules, Grafana dashboard queries, bats seams,
runbook filenames, homepage/AGENTS lore, and historical prose.
Rollback tags renamed in lockstep. Residue in the tree: zero.
Retired frank-medium/small/npu names are history and stay.
Known casualty: scripts/patches/*.patch context lines were rewritten;
those retired-era vLLM-fork patches no longer apply byte-exact.
The 2026-10-08 rename moved the harness model ids and the homepage cards;
the served aliases on the wire stayed frank-primary/frank-large/
frank-secondary. This closes the gap: the alias each endpoint announces is
now the family name, while the services, units, scripts, env files and
runbooks keep the old names (container identity stays stable).
- scripts/frank-secondary.env ALIAS, scripts/frank-large.env
HALOGEN_MODEL_ID, scripts/frank-primary.env FRANK_PRIMARY_SERVED_NAME
(+ the two serve-script defaults/gate messages)
- seams re-pinned: frank_secondary/frank_large/frank_primary unit bats
- fleet harnesses retargeted: bench-probe, deep-probe, calib-push,
fleet-probes-swift15, autoresearch(+measure), bench-threads-batch-http,
bench-decode
- shared/homepage/docs.html: api-id annotations and the "switchover will
rename the ids" footnote removed — the page's names are now the ids;
opencode/Pi/curl examples send frank-zealot
- dankclaw/hermes/config.yaml.example: frank-large -> frank-dragoon and the
live 262144 context (Hermes routes by served id; see Hermes.md)
Consumers with live state re-keyed out-of-band in the same window:
OpenWebUI (model row, access_grant, openai.api_configs model_ids) and the
Hermes config.yaml.
The panel title carries a right-aligned 'AI guide ->' link to
/docs#models (the running-models section), at the point of use.
Bare mode recolors the link; layout is flex baseline, doctrine-minimal.
Seam: homepage_config.bats 6/6; headless smoke census + screenshot.
- Icons: Copyparty -> improvised amber folder, Marreta -> crumbling
wall (bricks + red crack), Calibre-Web -> book, OpenWebUI -> official
logo.png from openwebui.com, Seerr -> official seerr.dev
os_logo_filled.svg.
- Tiles with an open link are clickable whole (.panel.clickable: hover
lift + glow; inner link clicks pass through).
- Any DOWN panel (app tile or model card) gets a red gradient wash and a
slow pulsing glow (.panel.down, .mi-card:has(.badge.down)).
- docs / admin / account-settings buttons moved from the footer to the
header (nav.top-links); footer keeps the deployment stamp.
- Zealot compaction advice promoted from the Use paragraph to a bordered
glowing model.compaction callout (.mi-comp); component seam pinned to
the new field.
- Every app card carries hex-clipped art: 19 amber-tinted brand marks
from the Simple Icons CDN plus 19 improvised monogram SVGs, wired via
a per-service "art" field; homepage_config.bats pins art presence for
all 32 apps.
- Model cards render zealot -> dragoon -> archon (registry order pinned
in the seam).
- Dragoon and zealot share one common uncensored legal warning (adult
content, harmful/hateful/illegal descriptions, sensitive offensive
cybersecurity topics, severe legal risk, "don't ruin this server for
everybody"), hover label on card and docs.
- Archon gets a bordered red permission callout: ask Frank before use.
- SearXNG obfuscation-not-anonymity note moved to a card-level hover
warning (.svc-warn) and a docs hover note.
- Gitea: self-service signup flow; no need to ask Frank.
- Footer docs/admin/account-settings become glowing bordered buttons;
the admin button is shown only to homepage-admins members via a new
Caddy /whoami endpoint echoing the outpost-injected X-Authentik-*
headers (spoof-proof; the /admin* 403 stays the real gate). e2e pins
both sides.
- Docs naming swept: Protoss names primary, bracketed api ids, explicit
note that a future switchover renames the ids themselves.
The model cards become the StarCraft trio: frank-large -> frank-dragoon,
frank-primary -> frank-archon, frank-secondary -> frank-zealot (family
names only; endpoint/API ids unchanged). The three cards fuse into one
Model Information component: per-unit Protoss accent colours and hex-
clipped unit art, 15px type, clickable StarCraft unit-voice quotations
(original .ogg files from the fandom wiki, served from /sfx/), supervision
notes (dragoon/archon safe for autonomous use, zealot tight supervision),
compaction-model recommendation for frank-zealot, and uncensored-model
legal notes rendered as hover tooltips instead of wall text. docs.html
carries the same renames plus api-id mapping.
App-card updates: Seerr, Calibre-Web, RomM, Jellyfin set to WIP: DOWN
with Open links removed (new wip "down" badge state); Copyparty notes it
serves all other media apps content including WIP apps; Gitea and Plex
noted as needing logins separate from SSO; SearXNG noted as obfuscated,
not anonymous (upstream API keys tie to Frank).
Plumbing: Caddy @data allowlist grew /sfx/*; deploy_homepage copies sfx/;
homepage_config.bats pins the new probe names, the component fields,
art/voice-file presence, and the wip-down schema.
Every fleet model card (and the #models docs section) now states, in
plain family-facing terms, what the model is equivalent to:
frank-large ~ Claude Sonnet 5 / OpenAI-Luna, frank-primary ~ Claude
Opus 4.8 / OpenAI-Terra (both Summer 2026 frontier), frank-secondary ~
last year (2025) frontier. Rendered as a new details.equiv line
(desc-equiv, styled in the head color, bare-mode clean); pinned in
homepage_config.bats.
- style.css: green phosphor -> amber CRT-AMBIOS palette; docs styles for
screenshots, code blocks, key warning (full + bare mode)
- index/admin/docs: VT323 from the Google Fonts CDN
- docs.html: rewrite #chat/#models for the live fleet (flash-server v2,
radiance native FP8, Qwen3.5-9B Q6_K); new #api-access with OpenWebUI
key walkthrough + Opencode and Pi configs; three OpenWebUI screenshots
- services.json: AI card details match the current fleet
- deploy-remote.sh: ship shared/homepage/img/ to /srv/homepage/
- docs: Homepage.md theme wave + OpenWebUI.md picker access-control law
Cross-cutting rename of the current llama.cpp endpoint on the RTX 3090
(port 7771) from frank-small to frank-secondary:
- Service trio: scripts/{frank-small.env,frank-small.service,
serve-frank-small.sh,download-frank-small.sh} → frank-secondary.*
- Test: tests/frank_small_unit.bats → tests/frank_secondary_unit.bats
- Runbooks: serverAppConfigs-docs/LLM/frank-small-*.md →
frank-secondary-*.md (9 files); the FreeToken runbook stays
freetoken-frank-small.md (it documents the retired CUDA-era service)
- Identifiers: FRANK_SMALL→FRANK_SECONDARY, llm_small→llm_secondary,
LLM_SMALL_*→LLM_SECONDARY_*, deploy_llm_small→deploy_llm_secondary
- Historical references to the OLD frank-small (FreeToken/CUDA on the
3090, and the Dankserver service that became frank-medium) are
preserved verbatim — they describe what the service was called
at the time.
Replace the FreeToken PR587 endpoint (single-GPU: its TP=2 dies in the
pynccl.cu ROCm JIT) with the radiance engine on both R9700s, TP=2 via
libr4d's fused one-shot all-reduce. Official Qwen/Qwen3.8-Flash-Next-FP8
payload at native block-FP8 precision:
- scripts/build-frank-primary-rad.sh: staging (quantization_config drop =
plugin SELECTION law) -> checkpoint reshape -> rad-convert with ONE
--quant rule (PLE rtn fixed-scale); plan-only first.
- scripts/stack-qwen38-official-fp8.py: the two value-preserving reshapes
(experts byte-stacked into the plugin's stacked layout; PLE's 128
scaleless fp8 shards decoded to one bf16 shard with the checkpoint's own
scale; the rtn rule re-encodes at that same scale - codes round-trip
exactly and land the one-element scale plane the plugin declares).
- scripts/hfget.sh + verified download (131/131 shards sha256-match).
- froggeric v22.5 chat template vendored (--override-chat-template).
- serve script (pinned ledger: 2x40 GiB pools + 47.7 GiB PLE + prefix host),
TTM preflight fixed for kernels with pages_limit only (no pages_max -
an absent knob is skipped, not read as zero); pages_limit raised to
52,428,800 pages = 200 GiB on the host pre-cutover.
- root unit, relinked; deployer gates image digest + .rad floor 158,000 MiB
+ health; prometheus re-adds the /metrics job.
- rollback: tag frank-primary-freetoken-pr587-20261005 (FreeToken trio +
Dockerfile); payload + image stay on disk until the window closes.
deploy-remote.sh also carries the parallel session's frank-small payload
gates (swift15) - committed here per Frank's direction to commit the tree.
FreeToken has no /metrics endpoint — the job scraped 404s forever, a
structural never-green, not an outage (the Lucebox DS4.1 precedent). The
LlmEndpointDown alert regex self-adjusts (frank-.* matches existing series
only). Comments updated; seam now asserts frank-primary's absence. If
FreeToken grows /metrics, re-add the job.
Replaces llama.cpp Gemma4-12B-QAT (payload + binary retained as rollback).
SGLang in a self-contained python3.12 venv with an extracted CUDA 13.0
nvcc toolchain for the sgl-kernel/flashinfer JIT.
Shape: fp8 KV, pinned 606,208-token pool / 4 x 151,552 slots, bf16 mamba
states pinned at 32 slots (the engine autofills the ssm pool to a fixed
~5.1 GB regardless of dtype), eager prefill (no headroom for graph
capture), no spec decode (the checkpoint ships no mtp.* tensors despite
mtp_num_hidden_layers=1 — NEXTN drafts with an uninitialised 4.24 GiB
ghost at 0% acceptance).
Verified: 83.9 tok/s decode, 170k single prompt, 4 concurrent 54k
requests, vision (in-checkpoint tower), reasoning_content field.
The stock SearXNG image ships a native kagi engine; enabling it needs an
API key that must never enter git, and SearXNG does not expand env vars
in settings.yml (the live secret_key literally was '${SEARXNG_SECRET}'
until now — the app tolerated the placeholder string silently).
- settings.yml: kagi engine enabled (weight 2, kg shortcut, paging off so
exactly one API credit per aggregated query, timeout 8.0 vs 5.0 scrapers
so a slow paid query is not a wasted credit); api_key stays ${KAGI_API_KEY}.
- new scripts/searxng-render-settings.py: deploy-time renderer, allowlisted
to KAGI_API_KEY/SEARXNG_SECRET/SEARXNG_BASE_URL, fails the deploy loudly
on an unresolvable placeholder, idempotent on redeploys.
- deploy-remote.sh copy_searxng_config runs the renderer against
/opt/searxng/.env after copying settings.yml into /opt/searxng/config.
- KAGI_API_KEY placeholder added to dankclaw/.env.example (real key staged
into the host-side /opt/searxng/.env, mode 600, out of band).
- bats: engine wiring, renderer substitution, loud-failure, deployer seam.
- docs: SearXNG.md Kagi section + renderer law; Providers.md carve-out on
the no-credentials policy (paid engine acceptable when bounded + ordered).
Verified live on dankclaw: direct API probe via the gluetun netns (200,
data.search[] shape), then an aggregated query returned 29/29 results
tagged kagi, zero unresponsive engines.
The 2026-09-26 same-day cutover that ended the Lucebox DS4.1 detour:
Dankserver's :7770 endpoint is now unsloth llama.cpp (the b11030-mix
prebuilt, ROCm gfx120X, bundled runtime) serving
unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 — 175.30 GiB across six sha256-
verified split parts — on both Radeon AI PRO R9700s, as a ROOT unit (the
frank-large shape: no container, no generated unit).
The deployed contract:
- Placement law: layer split -ts 1,1 with -ngl 999; the routed experts
and the 51B-param n-gram (PLE) table are COMPUTED in host RAM via
-ot exps=CPU,ngram_embeddings=CPU (the x4-pair measurement: streaming
over PCIe loses at 3.7 t/s, computing in RAM does not care).
per_layer_token_embd.weight=CPU is required by the shared MTP head's
borrow path (a GPU-resident borrow crashes the draft load).
mmap stays on; --cache-ram 16384.
- Spec chain draft-mtp,ngram-mod: the shared Q8_0 MTP head
(mtp-...-shared-Q8_0.gguf, 2,786,568,256 B) at depth 2, with the
draftless ngram-mod matcher chained behind it — the only legal way to
layer two spec implementations in one process.
- Vision explicit: --mmproj names the BF16 projector (this build never
autoloads; a server without it comes up healthy and text-only).
- Reasoning: --reasoning-format deepseek → reasoning_content; the
Qwen3.8 template's {xhigh, medium, low} reasoning_effort gate folded
by reasoningEffortMap in models.yml.
- One stream, the model's full 262,144 context; greedy server default
with Qwen-recommended sampling riding compat.extraBody.
The deployer's deploy_llm_primary is rewritten for the root unit: it
gates every artifact at its declared byte count (stat -c %s, never wc -c
— which read the whole 377 GiB lucebox file for minutes on 2026-09-26),
retires the lucebox user quadlet to dated home trash, waits out port
7770, relinks/enables/restarts the root unit and waits up to 1800 s for
/health. The quadlet-era tree under dankserver/frank-primary/ is
archived (dankserver/.AGENTTRASH/20260926-llamacpp-cutover/ on this
box; git history everywhere else) and its planner mapping replaced by
the scripts/ trio.
Observability flips back to the llamacpp namespace: a frank-primary
Prometheus job at 192.168.0.235:7770 (cross-host LAN — the netavark
published-port quirk that forces host.containers.internal for the two
Dankclaw endpoints is same-host only), the LlmEndpointDown runbook
text, and the dashboard description for three llama.cpp endpoints.
Docs: the new runbook (frank-primary-llamacpp-qwen38-flash-next-q8.md),
RETIRED banners on the lucebox and radiance runbooks and their index
entries, the AGENTS.md fleet section rewritten for the root-unit shape,
and the homepage frank-medium card renamed to frank-primary with the
Q8_0 Flash-Next spec (docs.html #models and the modality note updated:
images on frank-primary and frank-small, audio on frank-small only).
models.yml: the dankserver provider is "llamacpp (dankserver)" serving
frank-primary, contextWindow 262144, maxTokens 65536 (compaction-gap
law: 183,501 + 65,536 <= 262,144), input [text, image].
Suite: 346 green (frank_primary_unit.bats rewritten for the root unit —
16 tests pinning the placement law, the shared-head requirement, the
explicit projector, HIP env hygiene (BIND_HOST, never bare HOST), and
the deployer's byte-count gates; the quadlet-era planner and
changed-tests mappings rewritten for the trio; the observability scrape
test asserts three llamacpp jobs again).
Rollback: the lucebox image (localhost/lucebox-rocm:gfx1201) and the
378 GiB payload stay on the host; the Qwen payload stays fetched, so a
re-swap in either direction is a redeploy, not a refetch.
The frank-medium retirement tail the 2026-09-26 lucebox cutover left
uncommitted: the radiance quadlet tree, its swap/fetch tooling and its
seam test are deleted (archive copies live in dankserver/.AGENTTRASH/
20260926-frank-medium-radiance/ and scripts/.AGENTTRASH/20260926-
frank-medium-radiance/), the planner/allowlist/test-changed mappings
drop the service, and frank-large's chat template moves to scripts/
(its old home was the retired frank-medium dir).
Finished here:
- tests/frank_primary_unit.bats: the lucebox quadlet seam (boot
persistence, host-network bind, HIP_VISIBLE_DEVICES=0,1, HIP-reserved
env-name guard, checkout-mounted entrypoint/healthcheck, first-start
patience, deployer payload/port gates).
- test-changed.sh: explicit .AGENTTRASH/ case — retirement commits move
files to trash, and the wrapper refused those additions with "no
focused-test mapping".
- observability: the vllm: namespace left with the radiance runtime.
Rules keep only llamacpp:* series (prefix-cache reuse, spec-decode,
queueing, endpoint-down); the dashboard drops the vLLM KV/latency
panels; prometheus.yml documents that frank-primary (lucebox) has no
/metrics and therefore no scrape job. llm-fleet.json was left
corrupted mid-edit (missing opening brace) and is rewritten from HEAD.
Suite: 339 green (was 7 red).
Dim the spec line (--dim) so the card reads pale role -> dim spec -> bright
spring-green inputs -> dim use. The inputs line now pops as the bright middle
tier and the spec/use are de-emphasized as reference detail.
Add --input: #00ff9d to the palette and point .desc-inputs at it, so the
Inputs line is visually distinct from the bright-green spec (--fg), the
pale-green role (--head), and the dim-green use (--dim).
Each LLM card now lists its accepted inputs as a line (Inputs: text · image ·
audio), sourced from the harness models.yml:
- frank-large: text (no vision projector)
- frank-medium: text, image (Gemma4 vision path)
- frank-small: text, image, audio (Gemma4 projector audio encoder; audio is
not in models.yml's input list only because OMP's schema rejects it, but the
endpoint accepts input_audio parts)
New .desc-inputs CSS + bare-mode reset; /docs model guide gains an Inputs line
per model. bats 30/30 + live render verified.
Each LLM card now carries a details object (role/spec/use) rendered as a
bold pale-green role, a bright-green spec line, and a dim 'Use:' line.
app.js descHTML() falls back to the plain description for non-LLM cards;
new .desc-role/.desc-spec/.desc-use CSS + bare-mode reset.
Roles made neutral (no nicknames): frank-large = deep reasoning,
frank-medium = primary model, frank-small = fast utility model.
The /docs model guide was fixed — it had drifted to retired model info
(frank-large as Qwen3.8-Flash-Next 87.3 GiB/200k, frank-medium as FP8
~31 GiB) — now Qwen3.8-27B UD-Q8_K_XL 31.5 GiB/262k and MXFP4/W4A8
19.4 GiB, reordered large->medium->small. bats 30/30 + live render.
frank-large: cybersecurity + debugging (requires Frank's permission)
frank-medium: primary intelligent model (surpasses Claude Sonnet 5 per
Artificial Analysis) — software engineering, sysadmin, instruction tasks
frank-small: fast/cheap utility model — agentic search, RAG, research
subagents, translation, roleplay, summarization, small utility tasks
Updated both the status cards (services.json) and the /docs model guide.
Footer link opens the authentik default-user-settings-flow in a new tab,
giving users a self-service entry point to change their password and
manage MFA/credentials without admin intervention.
Frank
