Boot 2026-09-22 at 0.98 fill: KV pool auto-sized to 425,047 tokens (1.62x
the 262,144 window) with 14.42 GiB per card available for KV. The DFlash2
drafter on the Swift finetune measures mean acceptance length 7.00 / 85.7%
(easy) and 5.08 / 58.3% (varied) — it survives the base-model change.
Replaces the estimated ~420k / ~17.6 GiB figures with the measured values.
0.99 failed the vLLM startup memory check before weight load: the R9700
reports 31.86 GiB total but only 31.23 GiB free at device-init (the ROCm/HIP
context + NCCL + radiance custom all-reduce consume the 0.63 GiB gap), so
0.99 (31.54 GiB) exceeded free and the container crash-looped. 0.98 (31.22
GiB) is the ceiling and the prior production value.
Also fixed the runbook KV math that had shipped with the 0.99 change: the
~730k pool estimate was impossible (MXFP6's 25.16 GiB weights > MXFP4's
19.37 GiB, so the pool is smaller, not larger than MXFP4's 486k) and the
34.2 GiB/card figure was not the R9700 total. Recomputed ~420k tokens at 0.98
fill.
The serve script now passes --chat-template-file pointing at the
Qwen-Sharp v22.4.0 template. Without it the binary falls back to the
built-in peg-native template, which produces a ~73,600-token prompt for
a ~250-token request. With the sharp template wired in, the same request
produces a 62-175-token prompt.
Knowledge-base follow-up to a45f396. The runbook, both index one-liners, and
the AGENTS.md topology row all still declared FP8 active; flipped to
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq with unquantized bf16 KV (pool
486,464 tokens, full 262,144 window), and recorded the causal nuance: fp8 KV
was on both arms, so the FP8-vs-MXFP4 gap was weight precision, not KV - clean
KV is a quality hypothesis for W4A8, not a throughput one. No verdict yet.
The quadlet's "TEXT ONLY ... 400 here" comment was wrong: vision works on the
MXFP4 checkpoint (Qwen3_5ForConditionalGeneration carries the vision tower
through the Gemma4 merge path, unquantized) - verified by naming a solid red
48x48 PNG "Red". Comment corrected; harness models.yml input stays [text,image].
Docs-only; no re-deploy (the endpoint already serves vision).
The LLM backends section said two ROCmFPX endpoints; the fleet now serves
three (frank-large/medium/small) with different runtimes. Rewrite it, fix the
gotcha that claimed host.containers.internal does not resolve on a bridge
(it does — verified end-to-end for both on-host endpoints), and add a History
entry for the 2026-09-21 model-picker rebuild.
The per-key config DB already pointed web search at SearXNG (engine=searxng,
query URL http://192.168.0.143:8081/search), but only inside the data volume.
The quadlet now carries the same values as seeding env vars, so a fresh data
dir comes up wired, and the doc records the access path: the internal
host-published port, not the search. vhost — the Anubis->Authentik guard
chain is browser-only, and the bridge egress sits in SearXNG's pass_ip, so
the 4-req/IP/hour JSON cap never applies.
The first cutover crash-looped the unit with 'invalid split file name':
the fetch script concatenated the three split parts into a single GGUF,
but part 1's header carries ggml.file_count=3, so the byte-correct
concatenation still advertises itself as a split and the loader demands
a -NNNNN-of-NNNNN filename.
- serve-frank-large.sh: -m points at part 1; llama.cpp loads the split
natively (parts 2-3 auto-discovered in the same directory)
- deploy-remote.sh: LLM_LARGE_MODEL gate path follows
- download-frank-large-unsloth.sh: assembly removed; the parts ARE the
canonical payload, verified against live Hub LFS oids
- docs: runbook model section + troubleshooting note, AGENTS.md, log
- the redundant 87 GiB assembled file is staged on dankclaw at
~/.AGENTTRASH/20260921-frank-large-assembled/
Both pages now group their panels under section headings rendered by shared
renderSection()/renderPanel() components in app.js (SECTION_ORDER fixes each
page's order: user = AI, Web Apps, Media Library Apps; admin = Media Library
Apps, Observability, AI, Infrastructure). The WIP badge splits into WIP: OPEN
and WIP: CLOSED via the wip value ("open"/"closed"); the checker passes the
value through as wip-open/wip-closed. The frank-large card follows the
unsloth llama.cpp cutover, frank-medium follows the FP8 checkpoint, and the
/docs model guide matches. Tautulli (Plex statistics, 192.168.0.235:8181)
joins the admin page's Media Library Apps section with an open link.
Only the 8084 forward is exposed; the desktop GUI runs inside the
container netns and is internal-only. Drop the now-unneeded
CUSTOM_HTTPS_PORT (no host-port conflict exists inside the netns).
lscr.io/linuxserver/calibre:9.15.0 (desktop build) running calibre-server
as the s6 main service on internal 8090 -> host 8084, auth-gated with a
plaintext userdb (mode 600) bootstrapped once by the deploy handler.
The desktop GUI is kept on 8080/8182, basic-authed with the same
credential pair via the machine-local env file.
The verification looped over every LFS file under UD-IQ4_XS/ and MTP/ in
the live Hub tree — the five MTP head variants we do not fetch (BF16,
Q4_K_M, Q8_0, shared-BF16, shared-Q4_K_M) printed MISSING, set ok=False,
and set -e killed the script silently between verification and assembly
after a 3-hour download. Verify exactly the FILES set instead (passed to
python as argv), and report NOT IN HUB TREE when an expected file has
vanished upstream. Also fix the layout comment: the transport parts live
in UD-IQ4_XS/, not parts/.
New runbook frank-large-unsloth-qwen38-flash-next.md (runtime, payload,
deployed config, the explicit -md, the 114 GiB GTT memory model, client
contract, rollback). frank-large-ciru-orca.md marked RETIRED (rollback
target, payload and runtime intact on disk). AGENTS.md topology row,
frank-large section, harness note, runbook index, and the GPU-budget
lesson (GTT aperture ~114 GiB, not the stale 64 GiB rocm-smi figure —
the CIRU Orca held 92.6 GiB on the same device). LLM index header,
mermaid, active list, and the vault root index updated. Work-log entry.
Runbook deployed-flags table and AGENTS.md now name the template
(froggeric v22.4.0, repo-tracked, terse by default) and the rollback
target; the stale qwen-fixed-v22.3.jinja row and the long-gone
MM_MERGE_ENGINE note are gone.
node-exporter is net=host, so its 192.168.0.143:9100 bind is a host bind,
not a published port — the bridge-networked Prometheus container cannot
reach its own host's LAN IP there (netavark DNATs published ports only).
The node-dankclaw target has been red by construction since the
observability cutover while the container reported healthy. Point it at
host.containers.internal:9100 (netavark delivers that to the host LAN IP,
where node-exporter listens) — the same rule frank-large and frank-small
already use. node-dankserver is a different machine and keeps its LAN
address. Update the cross-file contract test and the Targets table.
DFlash2 (100.0 t/s decode, 287.1 t/s burst-4) beats MTP self-speculation
(85.9 t/s, 179.7 t/s) despite a lower accept length (4.24 vs 5.66): the
small separate drafter steps faster than the target's own MTP layer applied
autoregressively. DFlash2 is the deployed shape; both swap-script paths now
default to dflash/7. Docs updated.
- shared/prometheus/rules/llm-fleet.yml: the llamacpp reuse recording rule
divided cached by prompt, but llama.cpp's prompt_tokens_total EXCLUDES
cached tokens (its own HELP line), so the ratio was unbounded — measured
77 on a fully-warm frank-large, meaning LlmPrefixCacheCollapse could never
fire on that side. Now cached / (cached + prompt), with the traffic floor
counting both terms too; the header documents the unit trap. Also fixed the
LlmEndpointDown description: frank-large's CIRU llama.cpp build DOES export
the llamacpp:* family (verified live), so the alert covers all three
endpoints.
- shared/grafana/provisioning/dashboards/llm-fleet.json: re-cut for two
namespaces. Every panel reads both (the endpoint picker selects); the
KV-pool-fill panel retired 2026-09-16 returned with
vllm:kv_cache_usage_perc; the latency p95 panel returned with the vLLM
histograms (TTFT/ITL/e2e); the alerts panel is scoped to the fleet's
Llm.* rules; the stale halogen-era description is gone. All 21 panel
expressions validated against the live endpoints.
- runbook + log + Observability: prefix caching measured, not inferred —
reuse happens at 1,648-token block granularity (attention block size padded
to the mamba page size); an identical 13,021-token prompt went 2.30 s cold
to 0.68 s warm with 9,888 tokens (76 %) reused, while a 2,521-token prompt
registered zero hits. Rule counts corrected to 6 alerts + 3 recording rules.
./tests/run.sh: 325 pass.
The radiance TP=2 cutover changed the metric namespace frank-medium exports
(llamacpp:* -> vllm:*) and left two scrape targets addressing host processes
by an address a container cannot reach.
- shared/prometheus/rules/llm-fleet.yml: two namespaces side by side. Added
LlmRequestsWaiting, LlmKvCacheSaturated (vllm-only gauges llama.cpp never
had), the job:llm_vllm_prefix_cache_reuse:rate5m and
job:llm_spec_accept_length:rate5m recording rules, and re-cut
LlmPrefixCacheCollapse / LlmSpecDecodeOff to OR both families with
per-family traffic floors so an idle endpoint cannot fire them. Every
expression validated against live Prometheus.
- shared/prometheus/prometheus.yml: frank-small -> host.containers.internal:7771
and frank-large scraped at host.containers.internal:7770. Both are HOST
processes on Dankclaw and netavark DNATs published container ports only, so
the LAN-IP targets were red by construction; frank-large's CIRU llama.cpp
build does export the llamacpp:* family, contra the old comment.
- tests/observability_quadlet.bats: the two seams now encode the new contract
(per-endpoint reachability by process type; every named series must be on the
measured list for its engine; engine-independent rules must name both).
- homepage docs.html/services.json: frank-medium sees pictures, frank-large is
CIRU Orca with a single seat (was halogen w4b, two seats, 200k).
- docs: radiance runbook is ACTIVE (deployed flags, three-arm battery, verified
capabilities, rollback, replica-era sections labelled); llama.cpp runbook
carries the SUPERSEDED banner; Dual-R9700-Cooperation states the measured
verdict; Observability records the namespace split and the DNAT lesson;
AGENTS.md topology/frank-medium/Model Config/Benchmarking Policy/runbook
index flipped; both indexes updated; work-log entry added.
Also repaired agent-pass damage: prose prepended above frontmatter openers,
duplicated runbook bodies, a literal tool-call token pasted into a runbook, and
an index destroyed by diff artifacts (restored from HEAD, flips re-applied).
Vault link check: 128 pages, 0 unresolved. ./tests/run.sh: 325 pass.
Reanalysis after the 2026-09-20 swap to an ASUS ProArt B850-Creator WiFi
NEO: both R9700s now sit on CPU GPP root ports (x8/x8 Gen5 bifurcation),
every bridge reads AtomicOpsCap Routing+, and both endpoints read
AtomicOpsCtl ReqEn+ (CAP_EXP+28.w = 0x1440) — the exact precondition
Dual-R9700-Cooperation's test plan named. TP=2 is therefore unblocked,
not proven: no collective has been dispatched since the swap,
GGML_HIP_RCCL is still OFF, and the 272 t/s "tensor" number on record
measured a butterfly fallback.
- Dual-R9700-Cooperation: short answer rewritten, re-measured 2026-09-20
topology table, rebuild-hypothesis door marked moot, probe plan is now
standing work.
- Dankserver System Specs: Motherboard section; GPU BDFs 0b:00.0 ->
07:00.0, iGPU 10:00.0 -> 11:00.0, kernel 7.0.0-31, SMT on, and the new
eno2 (192.168.0.247) second 5GbE port holding the low-metric default
route while eno1 keeps the fleet's static .235.
- frank-medium-llamacpp-dflash2: rationale and hardware row split past
from present; dated work-log sections stay as history.
- AGENTS.md + both indexes: no longer assert a wall that no longer exists.
Docs only; frank-medium stayed active and healthy throughout (verified
/health). ./tests/run.sh 331/331.
The frank-medium runbook's deployed-flags table still carried the
predecessor's values (35,30 / 393216 / 196608 / draft-mtp / BF16 mmproj);
updated to the deployed shape (32,33 / 327680 / 163840 / draft-dflash / f16
mmproj) from scripts/frank-medium.env. The halogen runbook still described
itself as 'the active frank-large runbook'; marked RETIRED 2026-09-19 with a
pointer to [[frank-large-ciru-orca]].
The vLLM detour (a2b39b1, same day) is retired: the Q4_K_M llama.cpp setup
is the requested shape and the vLLM KV-quant options on sm_86 all measured
dead (fp8 hardware-impossible, int8 page-size unification, turboquant
mm-prefix). Root unit restored, vLLM quadlet/Dockerfile/fetch script moved
to .AGENTTRASH, runbook listed as same-day-reverted. Rollback target:
frank-small-vllm-gemma4.
Runtime swap on the RTX 3090 (port 7771):
- vLLM gemma-x86_64-cu129 dev build, digest-pinned, audio extra pip-installed
into localhost/vllm/vllm-openai:frank-small (Dockerfile, no COPY lines)
- google/gemma-4-12B-it-qat-w4a16-ct (10.26 GiB int4) + Google's bf16 MTP
assistant (845 MB) — the w4a16-ct assistant cannot draft in vLLM (drafter
path skips Marlin; acceptance 1.00), bf16 measures 2.80-3.56
- root systemd unit -> rootless Quadlet (dankclaw/frank-small/quadlet/);
manual CUDA device+lib mounts (no nvidia-container-toolkit on host)
- bf16 KV (fp8 hardware-dead on sm_86), 289,647-token pool, 2 concurrent
full-context sessions, CUDA graphs on
- native text+image+audio: the fleet's multimodal endpoint
- measured: pp@67k 1293 t/s, pp@213k 608 t/s, tg@213k ~13.6 t/s,
tg@85k ~17.6 t/s
- host driver 610.57.04 (Ubuntu 26.04); sm_86 is correct, no arch override
- planner/scoping: frank-small now selected by dankclaw/frank-small/ +
env/serve/download trio; old unit retired to scripts/.AGENTTRASH/
- docs: new runbook frank-small-vllm-gemma4.md; AGENTS.md, LLM index,
top docs index updated; old runbook moved to retired
--cache-ram caps the total bytes of saved prompt states in HOST RAM, and
alloc() refuses any single state larger than the whole cap. That refusal was
firing in production: 137 times in the two days from 2026-09-16, at 11.0-11.4
GiB per state, so every conversation past ~170k tokens was uncacheable and
re-prefilled from scratch on its next turn.
The state law at this shape is 68 KiB/token (16 full-attention layers x 4 kv
heads x 256 dim x 2 B x 2 for K/V, plus the MTP draft context's 4096 B), so
8192 MiB held ~120k tokens and 16384 holds ~240k: a FULL 196608-token slot
state (~12.8 GiB) now fits with ~3 GiB spare.
Host RAM only, allocated on demand and never pinned, so the VRAM fit is
untouched. New seam: the cap must clear one full slot's state and stay under
half the host's RAM (it fails on the old 8192 and passes on 16384). Also
corrects the adjacent claim that 0 is the build's default for
--ctx-checkpoints -- this binary's --help says 32.
- services.json: frank-large/medium/small move from admin-only to the user
page with current descriptions (halogen w4b iGPU; 35:30 layer-split MTP
27B on the R9700s; Gemma4-12B QAT vision path on the 3090) and docs
links to /docs#models
- docs.html: static user documentation (access chain, model guide, media
how-tos, troubleshooting); served at /docs via the post-auth homepage
listener; shipped by deploy_homepage; footer + index link to it
- tests: pages seam asserts the docs contract (page, link, Caddy route,
deployer copy)
- vault: Homepage.md synced (docs landing shipped, cards reclassified)
The two R9700s sit on unequal PCIe paths: ROCm0 (03:00.0) is CPU-root-port
Gen4 x16 (~31.5 GB/s); ROCm1 (0b:00.0) rides the B650 chipset through
bridges negotiating Gen4 x1 (~1.97 GB/s). With the MTP drafter, output
layers and heavier compute buffers pinned to the last-layers card, live
rocm-smi read 98% busy on the x1 card vs 40% on the x16 card. Moving one
DeltaNet layer (34) to ROCm0 shifts ~0.43 GiB off the bottlenecked card;
the seam fit lands 30.59/30.01 with gap 0.58 GiB, inside floor and slack.
36:29 fails both gates, so this lever is spent.
signal-cli 0.14.7 parses an all-digit -a value as a phone number (the
numeric account name never resolves), and the message is a -m option,
not a positional (a positional is parsed as a second recipient).
Verified live: corrected invocation returns type SUCCESS.
The bridge-subnet-only pair dropped the host's own health probe
(source 192.168.0.143/127.0.0.1) and failed a healthy deploy. The
deployer now installs the full set: ACCEPT 127.0.0.1, ACCEPT the host
IP, ACCEPT 10.89.0.0/16, then DROP.
169.254.1.2 is not a local address on this host (netavark runs its
bridges in aardvark-dns' netns), so the relay cannot bind it — and a
loopback bind refuses container traffic. Probe-socket proof: a host
socket on 192.168.0.143:18094 answers the container's 169.254.1.2:18094
dial while the LAN IP refuses from the container. The relay binds
192.168.0.143:18093 and the deployer installs an iptables INPUT pair
(ACCEPT 10.89.0.0/16, DROP rest) so the unauthenticated endpoint stays
off the LAN.
A socket bound to 127.0.0.1 refuses container traffic that arrives via
the podman bridge: the first live webhook failed with 'dial tcp
169.254.1.2:18093: connection refused'. The relay now binds the
host-gateway endpoint — link-local to the podman bridge, off the LAN —
and the deployer's health probe follows it there.
- runbook: identity now 4x262144, audio verified, throughput table with
the OOM and negative-result rows, deployed flags updated to reality.
- AGENTS.md frank-small paragraph matches the deployed shape.
- log.md: the sweep entry.
- scripts/frank-small.env: my earlier line-range edit had clobbered
NGL=999 and FLASH_ATTN=on out of the file; the seams caught it. The
live service was unaffected because the script's defaults match the
declared values exactly, but the file contract is restored.
Suite: ALL TESTS PASSED (321), vault 125 pages 0 unresolved.
The 393216/34:31/MTP deploy measured 30.78 GiB on ROCm1 against the
fit's 28.32 — the MTP draft buffers (one nextn layer's KV, 4096 B/pos,
plus draft buffers) came in 2.46 GiB above the estimate. Recalibrate
the seam's MTP constant to the measured 4.20 GiB total charge on card
1 and drop the floor 1.5 -> 1.0 GiB for this shape (ROCm1's 1.08 GiB
headroom is real; 35:30/36:29 fail card 0's floor at this pool, so
34:31 is the only split that fits 2x196k).
AGENTS.md and the runbook move to the MTP drafter: 34:31, 393216 pool
(196608 per conversation), the DFlash2 sidecar documented as a
one-line revert, and a dated part-2 section with the fit/measured
table and the recalibration.
Frank
