The root and LLM index pages still advertised two retired endpoints' pages as
live procedure:
- `frank-medium-radiance-mxfp4`: frontmatter said "ACTIVE frank-medium
runbook" and the body opened "The ACTIVE frank-medium deployment". Both
re-keyed to the era it ran (2026-09-21 -> 10-10), with [[frank-archon]]'s
Eras table named as the owner of that shape.
- `frank-medium-model-swap`: read as live how-to for a retired endpoint.
Re-keyed to historical, pointing at [[Model Swap and Rollback]] for the
current swap law.
- Both index pages' rows re-keyed to match (rows are hand-copied from
frontmatter - no generator - so both sites move together).
Refs: vault de-dup campaign, cluster C5.
The 2026-10-09 wire aliases become the service names everywhere:
systemd units, serve scripts, env files and FRANK_* variables,
deploy-plan/deploy-remote service keys, container names, Prometheus
job names + alert rules, Grafana dashboard queries, bats seams,
runbook filenames, homepage/AGENTS lore, and historical prose.
Rollback tags renamed in lockstep. Residue in the tree: zero.
Retired frank-medium/small/npu names are history and stay.
Known casualty: scripts/patches/*.patch context lines were rewritten;
those retired-era vLLM-fork patches no longer apply byte-exact.
Boot 2026-09-22 at 0.98 fill: KV pool auto-sized to 425,047 tokens (1.62x
the 262,144 window) with 14.42 GiB per card available for KV. The DFlash2
drafter on the Swift finetune measures mean acceptance length 7.00 / 85.7%
(easy) and 5.08 / 58.3% (varied) — it survives the base-model change.
Replaces the estimated ~420k / ~17.6 GiB figures with the measured values.
0.99 failed the vLLM startup memory check before weight load: the R9700
reports 31.86 GiB total but only 31.23 GiB free at device-init (the ROCm/HIP
context + NCCL + radiance custom all-reduce consume the 0.63 GiB gap), so
0.99 (31.54 GiB) exceeded free and the container crash-looped. 0.98 (31.22
GiB) is the ceiling and the prior production value.
Also fixed the runbook KV math that had shipped with the 0.99 change: the
~730k pool estimate was impossible (MXFP6's 25.16 GiB weights > MXFP4's
19.37 GiB, so the pool is smaller, not larger than MXFP4's 486k) and the
34.2 GiB/card figure was not the R9700 total. Recomputed ~420k tokens at 0.98
fill.
Knowledge-base follow-up to a45f396. The runbook, both index one-liners, and
the AGENTS.md topology row all still declared FP8 active; flipped to
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq with unquantized bf16 KV (pool
486,464 tokens, full 262,144 window), and recorded the causal nuance: fp8 KV
was on both arms, so the FP8-vs-MXFP4 gap was weight precision, not KV - clean
KV is a quality hypothesis for W4A8, not a throughput one. No verdict yet.
The quadlet's "TEXT ONLY ... 400 here" comment was wrong: vision works on the
MXFP4 checkpoint (Qwen3_5ForConditionalGeneration carries the vision tower
through the Gemma4 merge path, unquantized) - verified by naming a solid red
48x48 PNG "Red". Comment corrected; harness models.yml input stays [text,image].
Docs-only; no re-deploy (the endpoint already serves vision).
Runbook deployed-flags table and AGENTS.md now name the template
(froggeric v22.4.0, repo-tracked, terse by default) and the rollback
target; the stale qwen-fixed-v22.3.jinja row and the long-gone
MM_MERGE_ENGINE note are gone.
DFlash2 (100.0 t/s decode, 287.1 t/s burst-4) beats MTP self-speculation
(85.9 t/s, 179.7 t/s) despite a lower accept length (4.24 vs 5.66): the
small separate drafter steps faster than the target's own MTP layer applied
autoregressively. DFlash2 is the deployed shape; both swap-script paths now
default to dflash/7. Docs updated.
- shared/prometheus/rules/llm-fleet.yml: the llamacpp reuse recording rule
divided cached by prompt, but llama.cpp's prompt_tokens_total EXCLUDES
cached tokens (its own HELP line), so the ratio was unbounded — measured
77 on a fully-warm frank-large, meaning LlmPrefixCacheCollapse could never
fire on that side. Now cached / (cached + prompt), with the traffic floor
counting both terms too; the header documents the unit trap. Also fixed the
LlmEndpointDown description: frank-large's CIRU llama.cpp build DOES export
the llamacpp:* family (verified live), so the alert covers all three
endpoints.
- shared/grafana/provisioning/dashboards/llm-fleet.json: re-cut for two
namespaces. Every panel reads both (the endpoint picker selects); the
KV-pool-fill panel retired 2026-09-16 returned with
vllm:kv_cache_usage_perc; the latency p95 panel returned with the vLLM
histograms (TTFT/ITL/e2e); the alerts panel is scoped to the fleet's
Llm.* rules; the stale halogen-era description is gone. All 21 panel
expressions validated against the live endpoints.
- runbook + log + Observability: prefix caching measured, not inferred —
reuse happens at 1,648-token block granularity (attention block size padded
to the mamba page size); an identical 13,021-token prompt went 2.30 s cold
to 0.68 s warm with 9,888 tokens (76 %) reused, while a 2,521-token prompt
registered zero hits. Rule counts corrected to 6 alerts + 3 recording rules.
./tests/run.sh: 325 pass.
The radiance TP=2 cutover changed the metric namespace frank-medium exports
(llamacpp:* -> vllm:*) and left two scrape targets addressing host processes
by an address a container cannot reach.
- shared/prometheus/rules/llm-fleet.yml: two namespaces side by side. Added
LlmRequestsWaiting, LlmKvCacheSaturated (vllm-only gauges llama.cpp never
had), the job:llm_vllm_prefix_cache_reuse:rate5m and
job:llm_spec_accept_length:rate5m recording rules, and re-cut
LlmPrefixCacheCollapse / LlmSpecDecodeOff to OR both families with
per-family traffic floors so an idle endpoint cannot fire them. Every
expression validated against live Prometheus.
- shared/prometheus/prometheus.yml: frank-small -> host.containers.internal:7771
and frank-large scraped at host.containers.internal:7770. Both are HOST
processes on Dankclaw and netavark DNATs published container ports only, so
the LAN-IP targets were red by construction; frank-large's CIRU llama.cpp
build does export the llamacpp:* family, contra the old comment.
- tests/observability_quadlet.bats: the two seams now encode the new contract
(per-endpoint reachability by process type; every named series must be on the
measured list for its engine; engine-independent rules must name both).
- homepage docs.html/services.json: frank-medium sees pictures, frank-large is
CIRU Orca with a single seat (was halogen w4b, two seats, 200k).
- docs: radiance runbook is ACTIVE (deployed flags, three-arm battery, verified
capabilities, rollback, replica-era sections labelled); llama.cpp runbook
carries the SUPERSEDED banner; Dual-R9700-Cooperation states the measured
verdict; Observability records the namespace split and the DNAT lesson;
AGENTS.md topology/frank-medium/Model Config/Benchmarking Policy/runbook
index flipped; both indexes updated; work-log entry added.
Also repaired agent-pass damage: prose prepended above frontmatter openers,
duplicated runbook bodies, a literal tool-call token pasted into a runbook, and
an index destroyed by diff artifacts (restored from HEAD, flips re-applied).
Vault link check: 128 pages, 0 unresolved. ./tests/run.sh: 325 pass.
The runbook for the new deployment, plus the sweep of everything that still
described the vllm-radiance replica pair: the vault index and its ACTIVE
marker, the superseded banner on the radiance page, the pointer from the
ROCmFPX page, the homepage card, the Grafana dashboard description and the
observability page's scrape table.
The new page carries the two facts a future agent needs and the flags do not
say: llama.cpp's layer split is what lets BOTH cards serve ONE model without
any collective (vLLM's PP handoff is an RCCL send/recv, which is the primitive
this host cannot dispatch), and the rocBLAS Tensile-path trap that makes a
healthy-looking build fail on the first request.
The vault is the project's source of truth, so a link to a page that does not
exist is a promise the repo cannot keep. Three were dangling:
- Dankclaw/Gitea.md pointed twice at dankclaw/compose.yaml — **which this
session retired**. The clean cutover obligation from my own change: the page
now describes the unit that publishes 3000 and the canonical Caddy unit, and
records why the compose file is gone.
- Dankclaw/Hermes-Dashboard.md linked [[LAN DNS]]; the page is [[LAN-DNS]].
- LLM/frank-medium-radiance-mxfp4.md linked [[Observability on Dankclaw]]; the
page is [[Observability]].
scripts/check-vault-links.py + a seam keep it that way. The checker models
Obsidian's rules, which took three corrections to get right — and each was a
false positive that would have sent me editing healthy docs:
1. Wikilinks are case-insensitive. A case-sensitive matcher reported 45
breaks; 39 were `[[qBittorrent]]` vs qbittorrent.md.
2. `[[folder/Note]]` resolves by path suffix, not by stem.
3. `%20` in a markdown link is URL-encoding, not a filename.
Validating the linter against the resolver it models is the prerequisite for
trusting any of its output — the same discipline that caught a bad assertion of
my own earlier tonight.
Every measurement behind the frank-medium runbook was taken with scripts that
lived in /tmp on two hosts — reproducible only until the next reboot, and
invisible to anyone reading the repository. This moves the useful four into
scripts/, where this repo already keeps its bench harnesses:
bench-vllm-corpus.py code-shaped corpus from the repo's own sources, because
synthetic random prompts misreport draft acceptance
badly (2.9-3.6 tok/update against ~5.8 on code)
bench-vllm-serve.sh concurrency sweep; encodes the two traps that produced
silent garbage — the tokenizer argument, without which
the client dies resolving "frank-medium" on HF and
reports no numbers at all, and the --ignore-eos
semantics that must not be compared across
bench-turn-latency.py per-turn TTFT for an agent-loop shape; the instrument
that decided the routing policy, and the one that shows
cold-vs-warm prefill directly
bench-llm-soak.py sustained multi-family load; what found the capacity
wall (three 30k-token families, 75% cold re-prefills,
zero failures)
Each header carries its traps and the interpretation; the runbook's Benchmarks
section now points at them instead of describing methodology with no tooling.
Also adds the missing half of the script seam: scripts/*.py are compiled, not
just scripts/*.sh syntax-checked — a harness with a syntax error could otherwise
ship unnoticed until someone needed it during an incident.
Both python harnesses were smoke-tested against the live fleet before commit:
the corpus builder wrote 15 prompts, and the turn harness measured 12.5 s cold
against 1.4 s warm on the same prompt — the cache behaviour the runbook claims.
Both pages claimed there was no alerting; there are six rules now, so they say
which rules exist, that promtool and /api/v1/rules both confirm them, and — in
the same breath — that no rule reaches a person, because this stack has no
Alertmanager and no contact point. The dashboard's ALERTS panel is named as the
only place an alert becomes visible without opening Prometheus.
The capacity work in this runbook was done with ad-hoc probes; the same signals
now have history in Prometheus, so the page says which series to watch and
admits the honest gap — this stack has no alerting, so a saturated pool shows up
on a graph after the fact rather than paging anyone.
Two more hypotheses, both measured on the same three-family workload (one
~80k-token family plus two ~50k-token families, each turn re-sending its
document):
1. Long-prefix caching is fine. One card, idle engine, a ~50k-token prompt
sent three times: 22.3 s cold, then 1.4 s, 1.4 s — with a varying tail too
(22.4 / 1.41 / 1.41). So the engine's lifetime "Prefix cache hit rate: 29%"
is an average across every prompt that replica ever saw, not a verdict on
any one request, and the pinning policy is safe.
2. Serialising large requests does not fix the overload. Admission control
(at most one >16k-token request per card, small requests untouched) gave no
clear win on small samples — heavy 25.4 s -> 48.2 s median, mediums
34.7/45.2 s -> 22.7/22.6 s, cache-hit rates 14-33 % both ways — so it was
reverted like the placement change before it.
The arithmetic now closes: with three families on two cards, one card holds two
families and must carry 2 x 50k of cached prefix plus the 50k of the sequence
generating — 150k against 122,636 tokens. Those co-tenants evict each other
regardless of concurrency limits. The wall is capacity, and the runbook says so
with the numbers.
The obvious next lever for cache thrash was placement: send a new conversation
to the card carrying less context instead of alternating. Implemented it
(decayed prompt-token estimate per replica, continuations still pinned) and
A/B'd it against the counter on the workload it targets — one ~80k-token
family plus two ~50k-token families, each turn re-sending the same document:
counter heavy 13 turns median 25.4 s (23% cached)
medium 34.7 s / 45.2 s (14% / 15% cached)
balanced by load heavy 5 turns median 46.7 s (20% cached)
medium 23.4 s / 34.3 s (0% cached, small sample)
No improvement, so the change is reverted and routing.py keeps the counter.
The reason is arithmetic: a running request's live sequence and its cached
prefix draw on the same pool, so a card running two 50k-token generations has
already spent its 122,636 tokens before any cache entry is considered.
Placement chooses which card is over capacity, not whether one is.
Third hypothesis measured and rejected tonight (with SPEC depth kept and CHUNK
disproved); the runbook records all three so the next operator starts from
evidence rather than from these ideas looking obvious.
Rebooted dankserver unattended and verified without any manual step: all three
units active 59 s after boot, both replicas healthy ~3.5 min after boot, real
completion through :7770 — the failure mode this repo has been bitten by before
(a service that only ever ran from compose and stayed dark after a reboot).
Records why it works: [Install] WantedBy=default.target wired into the
generator's default.target wants, plus Linger=yes, plus a disk-backed compile
cache so a cold boot needs no warm-up beyond the model load.
A 25-minute soak with three concurrent large-context agent loops (each turn
re-sending a ~30k-token document, growing history) came back clean — 155
requests, 155 answers, no errors, no failover, both replicas steady — but only
25% of turns hit the prefix cache, the rest paying a 13.9 s median re-prefill.
That is the capacity budget showing: 122,636 KV tokens per card is about two
30k-token conversations with their cached prefixes, and a third family on the
same card evicts.
Chasing the obvious lever disproved it. Available KV per card is 5.44 GiB at
CHUNK=8192, 4.22 GiB at 16384 and 4.16 GiB at 4096 — where the engine refuses
to start. The budget is not monotonic in the chunk size, so 8192 is a sweet
spot rather than a compromise and freeing KV needs another card, not a knob.
Upstream ships dflash depth 7 as a content-dependent default measured for TP=2
on their corpus. Re-measured on one card here, two runs per setting, 16
repo-derived prompts at 512 output tokens:
SPEC=7 conc1 46.2 tok/s, TPOT 20.8 ms, 3.86 tok/update
conc4 118.7 / 127.0 tok/s
SPEC=5 conc1 41.5 tok/s, TPOT 23.1 ms, 3.41 tok/update
conc4 136.1 / 129.4 tok/s
Step time is nearly identical at both depths (~79 ms vs ~80 ms single stream):
7 wins by landing more tokens per step, 5 wins at conc 4 by ~8% because more
narrow verify steps pack better. Depth 7 stays — this fleet is interactive
agent loops where per-turn latency is what is felt — and the runbook now
records the numbers and the one-line switch for a batch-first workload.
Raising the prefill chunk to 16384 grows the arena enough that the KV cache
falls to 4.22 GiB against the 5.21 GiB a 114688-token context needs, so the
engine refuses to start. Measured single-card prefill at 8192 is ~2.5k tok/s
(~15k-token prompt, 5992 ms TTFT); the trade would mean cutting context for
unmeasured prefill gain, so the default stands.
The two replicas behind :7770 were on plain round-robin, which throws away
prefix-cache hits: an agent loop re-sends its system prompt and file context
every turn, and alternating replicas means every other turn pays a cold
prefill. Measured with 90k-token contexts, warm-turn TTFT was 2676 ms.
Pinning everything to one replica is not the answer either — a cold batch
(the shape a benchmark or a burst of parallel subagents produces) then loses a
card's worth of prefill: 16 requests took 30.1 s pinned against 18.6 s spread.
So the policy routes by request shape. A continuation (more than one message —
the agent-loop signature) is pinned to the replica that served its prefix;
anything else spreads across the cards by counter. Measured: 19.97 s for the
burst and 1095 ms warm-turn TTFT — the best of both, without a regression on
either workload.
The policy lives in proxy/routing.py, separate from the HTTP plumbing in
proxy/affinity-proxy.py, so it can be tested without aiohttp, a network or a
GPU. tests/frank_medium_proxy.bats covers it, and it earned its keep: the
first version hashed fresh prefixes to pick a card, which sends every member of
a same-prefix burst to one card (29.8 s, measured), and a stale index in the
pin path returned 500 on every continuation. Both are now assertions.
ROUTING=roundrobin and ROUTING=affinity stay available to re-measure the
choice on this host; the default is adaptive.
The radiance MXFP4 stack cannot run TP=2 on this host. RCCL's collective
kernels carry a hidden hostcall-buffer parameter, and ROCr refuses to
dispatch them on a device without PCIe atomics: the AQL dispatch fails with
hipErrorIllegalState ("the operation cannot be performed in the present
state"), which is what every two-rank collective hit. The second R9700 sits
behind the chipset bridges 05:00.0/06:09.0, which advertise
`AtomicOpsCap: Routing-` — a read-only capability, so no software can enable
atomic routing on that path and the driver's
pci_enable_atomic_ops_to_root() fails for that card. The first card, on CPU
root port 00:01.1, has atomics enabled.
Ruled out: kernel version (7.0.0-28/-30/-31 identical, after a full upgrade),
iommu=pt, P2P hardware (45.3 GB/s between the cards), the shipped librccl (a
from-source rebuild reproduces the failure at the same call path), vLLM and
pynccl (a bare two-rank torch.distributed allreduce fails the same way), all
NCCL transport knobs, and single-rank tests (NCCL short-circuits at
world_size=1, so one-card success proves nothing).
Deployment: three rootless Quadlets — frank-medium-a (:7771, card 1),
frank-medium-b (:7772, card 2), both TP=1 at 114688 context (262144 and
131072 do not fit one card's KV) with vLLM-profiled KV — and
frank-medium-proxy (:7770), a round-robin that preserves the fleet contract
of one endpoint and one served model name. frank-medium.container stays as
the TP=2 target for a host whose second card is atomic-capable; the deploy
handler does not install it and trashes any stale copy.
Measured through the proxy (vllm bench serve, random dataset, --ignore-eos):
conc 1 42.8 tok/s, conc 4 147.6 tok/s, conc 8 230.8 tok/s, 32k prefill at
~2.8k tok/s per card, TPOT p50 23.6-28.8 ms.
Also: healthcheck reads PORT so one probe serves all three units, the hermes
example and the harness models.yml drop to the real 114688 context, and the
runbook records the full evidence chain.
Frank