The serve script only passed --chat-template-kwargs, so the binary fell
back to the built-in peg-native template. A 250-token request produced a
73,600-token prompt (280x the expected ~260). Wiring in --chat-template-file
drops the prompt to 62 tokens. Draft acceptance 0.79 (mean 4.14), model
answers correctly, reasoning lands in reasoning_content.
The build default is 8192 MiB. dankclaw is memory-tight (frank-large's
70 GiB GTT plus the rest of the stack) and Gemma4 runs fully on the 3090
(-ngl 999), so the CPU-side cache is not the hot path. Cap it at 2048 to
give back ~6 GiB of LPDDR. Gemma4 is not a DeltaNet model, so the
>12.8 GiB CACHE_RAM floor does not apply.
Wired through frank-small.env (CACHE_RAM) -> serve-frank-small.sh
(--cache-ram), with a wiring seam so the cap cannot be silently unwired.
The first cutover crash-looped the unit with 'invalid split file name':
the fetch script concatenated the three split parts into a single GGUF,
but part 1's header carries ggml.file_count=3, so the byte-correct
concatenation still advertises itself as a split and the loader demands
a -NNNNN-of-NNNNN filename.
- serve-frank-large.sh: -m points at part 1; llama.cpp loads the split
natively (parts 2-3 auto-discovered in the same directory)
- deploy-remote.sh: LLM_LARGE_MODEL gate path follows
- download-frank-large-unsloth.sh: assembly removed; the parts ARE the
canonical payload, verified against live Hub LFS oids
- docs: runbook model section + troubleshooting note, AGENTS.md, log
- the redundant 87 GiB assembled file is staged on dankclaw at
~/.AGENTTRASH/20260921-frank-large-assembled/
Both pages now group their panels under section headings rendered by shared
renderSection()/renderPanel() components in app.js (SECTION_ORDER fixes each
page's order: user = AI, Web Apps, Media Library Apps; admin = Media Library
Apps, Observability, AI, Infrastructure). The WIP badge splits into WIP: OPEN
and WIP: CLOSED via the wip value ("open"/"closed"); the checker passes the
value through as wip-open/wip-closed. The frank-large card follows the
unsloth llama.cpp cutover, frank-medium follows the FP8 checkpoint, and the
/docs model guide matches. Tautulli (Plex statistics, 192.168.0.235:8181)
joins the admin page's Media Library Apps section with an open link.
lscr.io/linuxserver/calibre:9.15.0 (desktop build) running calibre-server
as the s6 main service on internal 8090 -> host 8084, auth-gated with a
plaintext userdb (mode 600) bootstrapped once by the deploy handler.
The desktop GUI is kept on 8080/8182, basic-authed with the same
credential pair via the machine-local env file.
The verification looped over every LFS file under UD-IQ4_XS/ and MTP/ in
the live Hub tree — the five MTP head variants we do not fetch (BF16,
Q4_K_M, Q8_0, shared-BF16, shared-Q4_K_M) printed MISSING, set ok=False,
and set -e killed the script silently between verification and assembly
after a 3-hour download. Verify exactly the FILES set instead (passed to
python as argv), and report NOT IN HUB TREE when an expected file has
vanished upstream. Also fix the layout comment: the transport parts live
in UD-IQ4_XS/, not parts/.
Replace the CIRU v4.4.1 Orca runtime with the unsloth llama.cpp prebuilt
(b11030-mix-5ff778e, ROCm gfx1151, bundles its own ROCm) serving
unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (87.3 GiB, three HF split parts
concatenated) on the Strix Halo iGPU. One conversation, 200,000 ctx,
PARALLEL_SLOTS=1, MTP depth 2 via the explicit -md shared-Q8_0 head
(sidecar auto-discovery does not search the mtp/ subfolder), text-only.
Fleet contract unchanged: port 7770, alias frank-large.
- serve-frank-large.sh: unsloth binary, -m/-md, --spec-type draft-mtp
--spec-draft-n-max 2, --jinja, preserve_thinking; gates on binary,
GGUF, MTP head; conditional --mmproj on ENABLE_VISION=1
- download-frank-large-unsloth.sh: resumable fetch of 3 parts + MTP head
+ mmproj, each verified against its LFS sha256 fetched live from the
Hub tree API; concatenates parts, records/verifies SHA256SUMS
- deploy-remote.sh: LLM_LARGE_MODEL/LLM_LARGE_DRAFT paths, pkill pattern
on the model path (not the binary name), wait_port_free gate
- tests: seams for 200k/1-slot, explicit -md, text-only, handler gates,
sweep safety
DFlash2 (100.0 t/s decode, 287.1 t/s burst-4) beats MTP self-speculation
(85.9 t/s, 179.7 t/s) despite a lower accept length (4.24 vs 5.66): the
small separate drafter steps faster than the target's own MTP layer applied
autoregressively. DFlash2 is the deployed shape; both swap-script paths now
default to dflash/7. Docs updated.
The entrypoint now reads the model's weight quantization from config.json
and serves the plain FP8 path (no MXFP4 patches, --quantization=fp8, MTP
self-speculation, ROCM_AITER unified attention with the R4D all-reduce)
or the MXFP4 patch-and-serve path. A model swap is now: change SNAP
(+ SPEC_METHOD/SPEC), restart.
- radiance-entrypoint.sh: quantization detection, gated MXFP4 env vars
and patch block, --quantization fp8 flag
- quadlet: SNAP -> Qwen3.8-27B-Uncensored-FP8, SPEC_METHOD=mtp, SPEC=8,
KV_MEM=0 (profile the shape, the documented plain-FP8 default)
- deploy-remote.sh: model-aware gates (read SNAP/SPEC_METHOD from the
quadlet; drafter gate only for dflash, r4d.so gate only for mxfp4)
- tests: quantization-aware drafter/spec and KV-pool tests
- hf-fetch.sh: generic, resumable, verified, idempotent HF fetch tool
- frank-medium-fp8.manifest: 17 files (10 config + 7 shards) with sizes
and LFS sha256
The conversation battery (scripts/bench-conversation.py, salted, same 10
prompts, 2026-09-20) measured three arms on the two R9700s:
arm I llama.cpp layer split 32:33 + DFlash2, Q8_0 geo decode 45.7 t/s, TTFT 0.72 s, burst-4 71.7 t/s
arm L llama.cpp tensor split 1:1 (RCCL), no drafter geo decode 30.5 t/s, TTFT 0.71 s, burst-4 54.3 t/s
arm R radiance vLLM TP=2 + DFlash2-FP8, MXFP4 geo decode 95.8 t/s, TTFT 0.21 s, burst-4 259.4 t/s
Radiance wins every axis, so it takes the alias. The board swap is what made
arm R possible: TP=2 was archived on 2026-09-16 because the second card sat
behind B650 chipset bridges with AtomicOpsCap: Routing-, and the ProArt
B850-Creator put both cards on CPU GPP root ports (Gen5 x8, Routing+, ReqEn+).
The R4D one-shot all-reduce needs no RCCL collective anyway.
- un-retire dankserver/frank-medium/: the TP=2 quadlet (banner rewritten —
it is the deployed shape now), radiance-entrypoint.sh, the healthcheck.
The replica pair and its affinity proxy stay archived; one process owns
both cards now.
- retire scripts/frank-medium.{service,env} + serve-frank-medium.sh and
bench-spec-ab.py to scripts/.AGENTTRASH/20260920-llama-layersplit/ as the
rollback path. build-llama-rdna4.sh and download-frank-medium-gguf.sh
stay: they are the tooling that rollback needs.
- deploy-remote.sh: deploy_llm_medium now gates on the image (pull once,
re-tag to the localhost name the quadlet resolves), the MXFP4 checkpoint,
the drafter, the pinned patch clone and the patched libr4d; retires the
llama root unit into the home trash instead of relinking it; then
install_quadlet + wait_container_healthy 1800s (first start compiles).
retire_vllm_medium_quadlets becomes retire_vllm_replica_quadlets — the
a/b/proxy trio only, since frank-medium is installed again.
- planner + test-changed map dankserver/frank-medium/** to frank-medium
and its seam; the scripts-trio mapping is gone with the files.
- tests/frank_medium_unit.bats rewritten for the container contract: boot
persistence of a GENERATED unit (no enable on this host's v1 generator),
TP=2 with the atomics-era NCCL workarounds kept out, the DFlash2 depth vs
the MXFP4 decode-kernel row band, the KV pin's coupling to the shape it
was calibrated at plus pool/VRAM arithmetic, the endpoint/tool/reasoning
contract, and the archive invariants.
- homepage card + docs updated, and they were stale twice over: they still
described the ajgazin Swift 35:30 / 393k payload, not the deployed shape.
Known regression, documented in the unit: vision. The MXFP4 checkpoint is
language_model_only and the launcher loads no projector, so images are a 400
here; the harness's vision role already routes to frank-small.
The warm-up request was a warm prefix-cache hit on the next run, so it
hid exactly what the arms were meant to differ on (prefill). Token
counts now come from the streamed usage of each job itself, and --salt
defeats cross-arm inheritance.
bench-conversation.py answers 'what does this endpoint feel like': ten real
jobs streamed once, TTFT split from decode tok/s, salted so no arm inherits a
warm cache, optional concurrent burst for capacity. Needed to compare the two
tensor-parallel runtimes the 2026-09-20 board swap unblocked.
dankserver-route-primary.sh pins eno1 (the fleet address, .235) as the
preferred default route after the board swap brought up eno2 with DHCP and a
lower metric; eno2 stays up as failover. nmcli device reapply, not connection
up, so the link this session rides is not torn down.
librccl DT_NEEDEDs librocm-core.so.1, so a GGML_HIP_RCCL=ON binary would not
exec from its bundled lib dir (measured: 'error while loading shared
libraries: librocm-core.so.1'). The layer-split build never pulled it in.
Layer split needs no collectives, so RCCL stayed OFF. The 2026-09-20 board
swap put both R9700s on CPU root ports with atomic routing, so a
tensor-parallel (row-split) runtime is now buildable: pass
GGML_HIP_RCCL=ON with a distinct BUILD_DIR beside the production one.
The CIRU launcher gates on hip/lib/libamdhip64.so + rocr/lib/libhsa-runtime64.so
under its runtime root and exits 2 without them; the build links the private
venv through its RUNPATH, so point those entries at the venv's unversioned
devel symlinks (idempotent ln -sfn). Without this the unit crash-looped for
the full 2400s health gate on first deploy.
Operator call 2026-09-19: switch the iGPU endpoint from
halogen-flash-server to jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca on the CIRU
v4.4.1 runtime with the QSA window hotfix (host build against the private
ROCm 10 SDK), one stream of the full 262144 trained window. The service
leaves the quadlet shape for a root unit (the same shape as frank-small and
frank-medium: a private-SDK host binary has no image to install and no
generated unit to dodge); the halogen quadlet is archived, the deploy
handler retires it on the host, gates on 7770, and its stray-run sweep
matches the model path so it cannot take frank-small down.
Operator call 2026-09-19: switch the target to
orcarouter/Qwen3.8-27B-Uncensored-GGUF Q8_0 (gated repo; the fetch script now
requires an HF token and verifies sha256, not just size), take the drafter to
the Q4_K_M rung of z-lab/Qwen3.8-27B-DFlash2-GGUF, cut the pool to two
163840-token conversations (327680, entirely inside the trained 262144
window), and balance the tensors across the cards (32:33) in place of the
35:30 x16-lean. The in-file MTP head stays one SPEC_TYPE change away.
The vLLM detour (a2b39b1, same day) is retired: the Q4_K_M llama.cpp setup
is the requested shape and the vLLM KV-quant options on sm_86 all measured
dead (fp8 hardware-impossible, int8 page-size unification, turboquant
mm-prefix). Root unit restored, vLLM quadlet/Dockerfile/fetch script moved
to .AGENTTRASH, runbook listed as same-day-reverted. Rollback target:
frank-small-vllm-gemma4.
int8_per_token_head: page-size unification fails (scale tensor breaks
the 256/512 head-dim ratio). turboquant_k8v4: backend validation fails
(no mm-prefix attention, which Gemma4's vision/audio path requires).
fp8: hardware-dead on sm_86. bf16 it is; history documented in
frank-small.env, seam pins auto + no backend pin.
The VLLM_ATTENTION_BACKEND env var does not exist in this vLLM dev
build (warns 'Unknown vLLM environment variable' and is ignored), so
the pin landed nowhere and Gemma4's hook force-selected TRITON_ATTN
again. Pass --attention-backend TURBOQUANT as a CLI flag instead.
Gemma4's config hook force-selects TRITON_ATTN for the mixed 256/512
head dims whenever no backend is selected, and TRITON_ATTN rejects the
turboquant KV dtype (measured: 'kv_cache_dtype not supported', crash
loop). The TURBOQUANT backend accepts any head size and the
turboquant_* dtypes, and the Gemma4 MTP speculator carries the
target's selected backend through to the draft layers.
int8_per_token_head keeps a separate fp32 scale tensor per (token,
head); Gemma4's hybrid head dims (SWA 256 / full 512) make the
per-token page bytes 4160 vs 8256, not an integer ratio, so
unify_kv_cache_spec_page_size raises NotImplementedError — nine
crash-loop restarts measured. TurboQuant k8v4 packs its block scales
inside the tensor, keeps the 2:1 page ratio, and has a dedicated
backend with an explicit Ampere path. 289,647 -> ~772k-token pool.
fp8 KV is hardware-dead on the 3090 (sm_86, no cvt.e4m3x2), so int8 is
the cheapest quant the TRITON_ATTN backend can run: pure ALU dequant,
same backend, no fp8 tensor cores needed. 289,647 -> ~579k-token pool,
two -> four full-context sessions. Quality verified against the bf16
baseline (needle-in-haystack at 213k + short battery) before the swap.
Runtime swap on the RTX 3090 (port 7771):
- vLLM gemma-x86_64-cu129 dev build, digest-pinned, audio extra pip-installed
into localhost/vllm/vllm-openai:frank-small (Dockerfile, no COPY lines)
- google/gemma-4-12B-it-qat-w4a16-ct (10.26 GiB int4) + Google's bf16 MTP
assistant (845 MB) — the w4a16-ct assistant cannot draft in vLLM (drafter
path skips Marlin; acceptance 1.00), bf16 measures 2.80-3.56
- root systemd unit -> rootless Quadlet (dankclaw/frank-small/quadlet/);
manual CUDA device+lib mounts (no nvidia-container-toolkit on host)
- bf16 KV (fp8 hardware-dead on sm_86), 289,647-token pool, 2 concurrent
full-context sessions, CUDA graphs on
- native text+image+audio: the fleet's multimodal endpoint
- measured: pp@67k 1293 t/s, pp@213k 608 t/s, tg@213k ~13.6 t/s,
tg@85k ~17.6 t/s
- host driver 610.57.04 (Ubuntu 26.04); sm_86 is correct, no arch override
- planner/scoping: frank-small now selected by dankclaw/frank-small/ +
env/serve/download trio; old unit retired to scripts/.AGENTTRASH/
- docs: new runbook frank-small-vllm-gemma4.md; AGENTS.md, LLM index,
top docs index updated; old runbook moved to retired
--cache-ram caps the total bytes of saved prompt states in HOST RAM, and
alloc() refuses any single state larger than the whole cap. That refusal was
firing in production: 137 times in the two days from 2026-09-16, at 11.0-11.4
GiB per state, so every conversation past ~170k tokens was uncacheable and
re-prefilled from scratch on its next turn.
The state law at this shape is 68 KiB/token (16 full-attention layers x 4 kv
heads x 256 dim x 2 B x 2 for K/V, plus the MTP draft context's 4096 B), so
8192 MiB held ~120k tokens and 16384 holds ~240k: a FULL 196608-token slot
state (~12.8 GiB) now fits with ~3 GiB spare.
Host RAM only, allocated on demand and never pinned, so the VRAM fit is
untouched. New seam: the cap must clear one full slot's state and stay under
half the host's RAM (it fails on the old 8192 and passes on 16384). Also
corrects the adjacent claim that 0 is the build's default for
--ctx-checkpoints -- this binary's --help says 32.
Isolated-port (:7790) 2k-token decode with AVG rocm-smi util over the
request (peak util lies: a layer split touches both cards every token):
35:30 draft ROCm1 31.52 t/s util0 41% util1 98% <- deployed
36:29 draft ROCm1 31.49 t/s util0 43% util1 98%
35:30 draft ROCm0 31.52 t/s util0 42% util1 98%
36:29 draft ROCm0 31.48 t/s util0 43% util1 98%
The x1 card (ROCm1) stays the 98% bottleneck in every configuration.
Moving the MTP drafter to the x16 card does not flip the balance: the
per-step cost is the verify phase (both cards run the full target) plus
the logits, which are pinned to the card holding the last layers. A
target-layer move (36:29) buys 2 points on the x16 card and erodes its
VRAM margin (fit free 0.45 GiB) for no throughput change. The x1 card is
compute-bound, not PCIe-bound (~5.2 MB activation per ubatch on the link),
so saturating it costs no tokens. 35:30 is the balance; no split change.
A pending entry for frank-default-sightsound (FLM NPU endpoint retired
2026-09-11) survived in the dankclaw state store and made every --pending
run exit 2 at the dispatcher before the Playwright e2e gate could
execute. The pending pass now drops entries the target no longer allows
into ~/.AGENTTRASH and keeps going. Behavioral seam test added.
- services.json: frank-large/medium/small move from admin-only to the user
page with current descriptions (halogen w4b iGPU; 35:30 layer-split MTP
27B on the R9700s; Gemma4-12B QAT vision path on the 3090) and docs
links to /docs#models
- docs.html: static user documentation (access chain, model guide, media
how-tos, troubleshooting); served at /docs via the post-auth homepage
listener; shipped by deploy_homepage; footer + index link to it
- tests: pages seam asserts the docs contract (page, link, Caddy route,
deployer copy)
- vault: Homepage.md synced (docs landing shipped, cards reclassified)
The two R9700s sit on unequal PCIe paths: ROCm0 (03:00.0) is CPU-root-port
Gen4 x16 (~31.5 GB/s); ROCm1 (0b:00.0) rides the B650 chipset through
bridges negotiating Gen4 x1 (~1.97 GB/s). With the MTP drafter, output
layers and heavier compute buffers pinned to the last-layers card, live
rocm-smi read 98% busy on the x1 card vs 40% on the x16 card. Moving one
DeltaNet layer (34) to ROCm0 shifts ~0.43 GiB off the bottlenecked card;
the seam fit lands 30.59/30.01 with gap 0.58 GiB, inside floor and slack.
36:29 fails both gates, so this lever is spent.
signal-cli 0.14.7 parses an all-digit -a value as a phone number (the
numeric account name never resolves), and the message is a -m option,
not a positional (a positional is parsed as a second recipient).
Verified live: corrected invocation returns type SUCCESS.
Every webhook died on 'cannot access local variable _last_sent' because
the prune comprehension rebound the module global inside the function.
Prune in place instead.
The bridge-subnet-only pair dropped the host's own health probe
(source 192.168.0.143/127.0.0.1) and failed a healthy deploy. The
deployer now installs the full set: ACCEPT 127.0.0.1, ACCEPT the host
IP, ACCEPT 10.89.0.0/16, then DROP.
169.254.1.2 is not a local address on this host (netavark runs its
bridges in aardvark-dns' netns), so the relay cannot bind it — and a
loopback bind refuses container traffic. Probe-socket proof: a host
socket on 192.168.0.143:18094 answers the container's 169.254.1.2:18094
dial while the LAN IP refuses from the container. The relay binds
192.168.0.143:18093 and the deployer installs an iptables INPUT pair
(ACCEPT 10.89.0.0/16, DROP rest) so the unauthenticated endpoint stays
off the LAN.
A socket bound to 127.0.0.1 refuses container traffic that arrives via
the podman bridge: the first live webhook failed with 'dial tcp
169.254.1.2:18093: connection refused'. The relay now binds the
host-gateway endpoint — link-local to the podman bridge, off the LAN —
and the deployer's health probe follows it there.
- runbook: identity now 4x262144, audio verified, throughput table with
the OOM and negative-result rows, deployed flags updated to reality.
- AGENTS.md frank-small paragraph matches the deployed shape.
- log.md: the sweep entry.
- scripts/frank-small.env: my earlier line-range edit had clobbered
NGL=999 and FLASH_ATTN=on out of the file; the seams caught it. The
live service was unaffected because the script's defaults match the
declared values exactly, but the file contract is restored.
Suite: ALL TESTS PASSED (321), vault 125 pages 0 unresolved.
Swept on the 3090 at the deployed 4x262144 geometry, salted API probes:
- ubatch 512: pp@254k ~918 t/s, tg@250k 90-204 (noisy)
- ubatch 1024: pp@254k 962 t/s (+4.8%), tg@250k 257-302
- ubatch 2048: OOM — the MTP draft context needs ~3.3 GiB of pp compute
buffer against 2.2 GiB headroom; do not try.
- draft depth 4 + poll 100 decoded WORSE at 250k (230-257) than 3/50:
the longer verification round re-reads more long-ctx attention than the
extra accepted token earns. Kept 3/50, now with the evidence attached.
Idle VRAM 23,300 MiB; all ubatch-1024 variants idled identically
(compute buffers reserved at max graph size), so 8192 buys the deepest
prefill window for free over 4096.
Both were implicit build defaults (3 and 50); the throughput sweep needs
to move them without editing the exec line each time. Defaults preserve
current behavior exactly. Additive only.
DFlash2's sidecar (1.92 GiB + buffers on ROCm1) kept the 393216 pool off the table; the MTP head lives in the target file, so its cost is one nextn layer's KV (4096 B/pos) plus small buffers. Fit: ROCm0 29.77 / ROCm1 28.32 GiB, over the floor, 34:31 the balance point at this pool. n_max 3 is the measured MTP winner. The seam's fit is now drafter-aware, the slack is the deployed integer gap (1483 MiB), the ctx cap 393216 with the rope caveat (top 131k extrapolates on the base rope).
CTX_SIZE=1048576 / PARALLEL=4: every slot gets the model's full trained
window. Measured fit arithmetic on the 3090: 1 full-context slot idles
at 11,996 MiB, 4 at 22,346 -> 3,450 MiB per slot, ~8,546 MiB fixed.
A 5th slot projects 25.8 GiB past the 24,576 MiB card, and the KV pool
allocates upfront, so 4 is the ceiling. Explicit CTX_SIZE bypasses --fit
on purpose: fit maximises the TOTAL pool and would divide a smaller
window across slots at parallel>1. The slot-grid seam now pins both the
full-window division and the budget inequality so a future capacity
bump fails pre-deploy, not in the boot allocation.
Audio verified while measuring: the mmproj's audio encoder transcribes
input_audio parts verbatim (JFK WAV sample). Header comments updated.
The Qwen3.8-27B IQ3_S file carried its own MTP head; Gemma4's ships as a
separate assistant file (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant,
4 nextn layers) that loads via --model-draft and shares the target's last
layers' KV. The draftless ngram-mod matcher chains behind it in the same
--spec-type value — two context-backed drafters cannot share one process,
so this chain is the only shape that carries both.
KV is q8_0 (16 KiB/token/layer vs 32 at f16) and CTX_SIZE is unset so
--fit sizes the pool against what the 7.4 GiB of payloads leave on the
24 GiB card. build-cuda was rebuilt with -DGGML_CUDA_FA_ALL_QUANTS=ON,
which quantized KV on CUDA requires. The three seams that pinned the Qwen
contract (MTP-in-file, the Qwen mmproj name, the f16 KV arithmetic) now
pin the Gemma4 contract instead, and the deploy handler's presence check
points at the new payload.
The 36:29/327680/vision deploy came up healthy and text-only: image requests
answered 500 'image input is not supported - hint: ... provide the mmproj'
with no mmproj line in the load log, so llama-server fb27a525 does not
autoload mmproj*.gguf from the model directory. Name the file, the way
frank-small's serve script already does, and pin the flag in the seam.
Frank
