The target moves from unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL to
ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF's UD-Q8_K_XL —
an abliterated fine-tune of the same base, under the non-OSI SWIFT OPEN
LICENCE 1.0. Port, alias, context, drafter and client contract are
unchanged.
Compatibility was settled from the GGUF headers before fetching anything:
both files parse to the same architecture (qwen35), the same 866 tensors
with identical names in identical order, zero per-tensor ggml_type or dim
mismatches, and the same MTP head. Two metadata differences that look
alarming are no-ops, confirmed against gufo's source — general.file_type is
deliberately not consulted (the file gufo already served records Q4_K_M
while containing no Q4_K tensor), and Swift's extra
qwen35.attention.recurrent_layers key is never read; layer kinds come from
full_attention_interval, which both files carry as 4 and which agrees.
The load test agreed: 9.2 s to listening, 66.5 GiB of the 116.7 GiB iGPU,
no loader warnings, and a green contract battery including a two-turn tool
loop.
The DFlash2 drafter did not move, so the payload is now two files in two
directories: DRAFTER_DIR is its own tunable. That broke the deployer's
stray-run sweep, which keyed on the target filename and would have stopped
matching — a stray run would then have held 7770 past the port gate. It
now keys on the drafter, which every frank-large command line carries.
The fetch script sends no auth header: the repo is public, and a token on
the curl command line is world-readable via /proc/<pid>/cmdline. The
anonymous transfer ran at 1.8 MiB/s; the host token through a mode-600
curl -K config reached 7 MiB/s.
Test seam: the mutation pass found four real gaps. Two fetch assertions
were satisfied by a header comment rather than the URL assignment, so
repointing a repository left them green, and nothing bound the env's
MODEL_FILE/DRAFTER_FILE to the filenames the fetch scripts download. Both
are now explicit. A harness bug worth recording: a bats --filter that
matches no test reports zero "not ok", indistinguishable from a mutation
that survived — the harness now checks the 1..N count. 16/16 caught.
The ad-hoc status commands kept producing wrong readings, most seriously
reporting "MemAvailable 0.0 GiB" on a host that actually had 240 GiB free.
Cause: /proc/meminfo separates key from value with a TAB, so
grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2
returns an empty field and awk prints 0.0. A false "host is out of memory"
reading is worse than no reading, because it invites an unnecessary abort.
dankstatus.sh parses on a colon split (tab-agnostic), cross-checks
MemAvailable against MemTotal before printing either, and refuses to print
a zero value. It also pins the rocm-smi CSV column order (device, total,
used) which was previously read swapped, and takes only the first line of
systemctl is-active/is-enabled, which emit more than one line here.
Incidental finding while validating: rocm-smi reports card2 (the 7700X iGPU)
as 0.50 GiB total, not the 123.0 GiB torch.cuda reports. torch was exposing
the GTT aperture. The device is smaller than documented, not larger.
Tests: 346/346 pass (no test surface touched).
Live numbers from the 2026-09-28 deploy: 13 s load to listening, RSS
30.6 GiB, ttft 549.8 ms / inter-token 24.2 ms on a short prompt, and a
green contract battery (health, ready, /v1/models at 262,144, content +
reasoning_content, tool_calls, SSE, and a two-turn tool loop).
Also recorded: gufo accepts every request field the harness sends in
extraBody and the full reasoning_effort set, both verified 200 — which
is why models.yml's reasoningEffortMap fold is now optional rather than
required, and why it is being kept anyway.
gufo is a vertical inference engine built for exactly this hardware
(Ryzen AI MAX+ 395 / Radeon 8060S, gfx1151), and Dankclaw's frank-large
has swapped runtimes seven times in search of a better one. This one
ships no prebuilt Linux binary, so the runtime is BUILT FROM SOURCE on
the host and the payload, port, alias and client contract stay exactly
where they are.
Runtime: gufo v0.1.1 (b0f8673) at /home/dank/GitPackages/gufo-v0.1.1,
built by the new scripts/build-gufo.sh. It links the host ROCm 7.2.4
through its RPATH — the opposite of the unsloth prebuilt it replaces,
which carried its own — so two host workarounds are encoded in that
script: Ubuntu's librocwmma-dev installs a truncated rocWMMA header set
that shadows AMD's and cannot be overridden with -I in HIP mode, and the
ROCm clang prefers Ubuntu 26.04's GCC 16 headers over the GCC 15 whose
libstdc++ is actually installed.
Shape is unchanged: one session, full 262144 ctx, the same UD-Q8_K_XL
target and the same DFlash2 drafter, greedy, text-only. Speculation
moves from llama.cpp's fixed depth 4 to gufo's adaptive block policy
capped at 7 proposals.
Client-contract deltas, all documented: the chat template is compiled
into the binary (no --chat-template-file, so the Qwen-Sharp 73k-token
failure mode is gone), reasoning_effort now takes the full OpenAI set
(so models.yml's reasoningEffortMap fold is no longer required — kept
anyway, so this commit does not also move every high role from xhigh to
high effort), and there is no KV-quant or prompt-cache-RAM flag.
The test seam now runs the serve script against a stub engine and
asserts on the argv it actually execs, instead of grepping the script's
text. The deploy handler gates on the runtime binary and returns before
it stops the serving unit.
Measured trade, from gufo's own Q8_K_XL tables against llama.cpp
b11069: prefill +37% to +80%, cold load 4.5x faster, decode +7% at zero
depth — but -14% at 65,536 tokens of depth and -24% at 131,072. On the
8-bit target the crossover is ~32k, and frank-large runs one 262,144
conversation. The Q4_K_XL target shows no such regression.
Answers Frank's question: given the 2x R9700 box, which of DeepSeek-V4.1-Flash,
MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash runs best, on what runtime, and can
Affinity / vllm-radiance / FreeToken-AMD be modified to run them fast.
Sizing, measured from the HF API rather than the model cards:
- DS4.1 payload 475.27 GiB, of which 189.13 GiB is Engram (two 94.56 GiB
shards, the layer 1 and 14 lookup tables). Lucebox never maps them, so
they do not evict the expert page cache.
- 240 routed experts x ~11 MB per token at the 37-of-384-per-layer residency
the hot stack can hold = ~2.4 GB across PCIe per token. At 55 GB/s real on
Gen5 x8 that is a ~23 tok/s ceiling, under the 35 tok/s goal before any
attention or dense work.
- The backend refuses layer-major prefill for V4.1, so the only legal
prefill is token-by-token; 1000 t/s is ~20x away.
Affinity (Codeberg StillDeadcode) builds clean here (hipcc 23, gfx1201, zero
errors) and claims 110 tok/s for the 284B V4-Flash, because the 101 GiB
container leaves the cold expert set in a pinned host pool in RAM. V4.1 is
2.5x too big to repeat that. Port scope, from both checkpoints' real
safetensors indices: expert machinery ports unchanged (identical tensor
naming, 15,360 tensors at 40x384), 26/32 per-layer tensors and 34/35 config
keys present. Five gaps - Engram, the disjoint {0,1,2} vs {0,4,128} ratio
vocabulary, CED, the DSpark draft geometry, vision. num_hash_layers is the
only key V4.1 lacks, so model.h:36's default of 3 silently survives and
would route layers 0-2 through a table that does not exist.
Also corrects: Dankserver RAM 60 -> 246 GiB (stale since the 2026-09-20 board
swap) and storage; the top-level index credited V4.1 with V4's geometry
(43 layers / 256 experts vs the actual 40 / 384).
New ACTIVE runbook (setup/doctor PASS=34/FAIL=0, UVA transport verified
in serving workers, wire-verified alias/reasoning/metrics), index +
vault entries flipped: llama.cpp b11030 endpoint RETIRED 2026-09-27 to
rollback target. Topology diagram updated.
The stock SearXNG image ships a native kagi engine; enabling it needs an
API key that must never enter git, and SearXNG does not expand env vars
in settings.yml (the live secret_key literally was '${SEARXNG_SECRET}'
until now — the app tolerated the placeholder string silently).
- settings.yml: kagi engine enabled (weight 2, kg shortcut, paging off so
exactly one API credit per aggregated query, timeout 8.0 vs 5.0 scrapers
so a slow paid query is not a wasted credit); api_key stays ${KAGI_API_KEY}.
- new scripts/searxng-render-settings.py: deploy-time renderer, allowlisted
to KAGI_API_KEY/SEARXNG_SECRET/SEARXNG_BASE_URL, fails the deploy loudly
on an unresolvable placeholder, idempotent on redeploys.
- deploy-remote.sh copy_searxng_config runs the renderer against
/opt/searxng/.env after copying settings.yml into /opt/searxng/config.
- KAGI_API_KEY placeholder added to dankclaw/.env.example (real key staged
into the host-side /opt/searxng/.env, mode 600, out of band).
- bats: engine wiring, renderer substitution, loud-failure, deployer seam.
- docs: SearXNG.md Kagi section + renderer law; Providers.md carve-out on
the no-credentials policy (paid engine acceptable when bounded + ordered).
Verified live on dankclaw: direct API probe via the gluetun netns (200,
data.search[] shape), then an aggregated query returned 29/29 results
tagged kagi, zero unresponsive engines.
Instrument the live server to answer "can -ot target the most active
layers": AMD IBS op sampling on llama-server during 4 mixed concurrent
generations, sampled load addresses mapped through /proc/<pid>/maps to
backing tensors with extents derived from consecutive GGUF header
offsets (no quant type-size tables - a wrong per-element size silently
mis-buckets tensor tails, first run lost 59% of expert samples to a
junk bucket).
Measured (75 s window, 2.7M expert-tensor samples): per-block read
volume is FLAT, 0.93x-1.08x of mean across all 40 CPU-resident blocks.
MoE top-k routing fires every block every token, so "most active
layers" do not exist at -ot granularity - any 8 GPU-resident blocks
are equivalent, the lever is COUNT, already at the 5/3 VRAM ceiling.
Real skew is per-EXPERT (top 10% of 48x512 cells carry 40.5% of
samples), below tensor-override granularity. No config change.
scripts/frank-primary-expert-activity.py: dump + sample + analyze in
one entry point; --analyze-only re-analyzes saved samples. Instruments
production serving in place; not a benchmark.
29,696 B/token ctx-scaled (12 of 48 layers carry KV - full_attention_interval=4;
indexer and draft KV included); weights/RS/compute are ctx-independent.
262k = 18.75 GiB total / 9.37 per card; 100k = 14.3 / 7.1. Parallel
slots split the pool rather than multiplying it.
Battery 6/6: alias, reasoning_content, tool_calls, vision (probe image
described exactly), reasoning_effort high -> 200. Decode 11.8 t/s at
depth-0 prose (draft 109/accepted 72), 0.864 mean len 2.73 on tool
traffic. VRAM 29%/36% (output-head card carries the drafter). Also
records the ROCm1 drafter law in the runbook body (borrowed output.weight
+ n_copies==1 abort) and the two cutover traps: foreign-process health
false green, and llama-server SIGTERM cleanup hang.
Discovery, not guesswork: dump the GGUF v3 headers of the fetched parts
(scripts/dump-gguf.py, added — a dependency-free reader that never
touches tensor data, so a 50 GiB part scans in milliseconds) before
the first load.
What the dump says (parts 2-3, 2026-09-26):
- The 51B-param PLE n-gram table is ONE tensor: per_layer_token_embd.
weight, dims [160, 320001536], Q8_0, 54.4 GiB — exactly the whole of
part 3. It is the same tensor the shared MTP head borrows, so the
=CPU override is required for placement AND for the draft load
(GPU-resident borrow: "invalid vector subscript").
- The model is qwen4exp, 512 experts, top-10 routing
(qwen4exp.ple.ngram_size=3, heads_per_ngram=8).
- Part 1 is metadata-only (ntensors=0); parts 2-6 carry tensors.
The placeholder pattern ngram_embeddings matched nothing — dropped from
OT_OVERRIDES (env, serve-script default, seam, AGENTS.md, runbook,
LLM index) before it could masquerade as a working override at load
time. Deployed set: exps=CPU,per_layer_token_embd.weight=CPU.
Suite: 346 green.
The 2026-09-26 same-day cutover that ended the Lucebox DS4.1 detour:
Dankserver's :7770 endpoint is now unsloth llama.cpp (the b11030-mix
prebuilt, ROCm gfx120X, bundled runtime) serving
unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 — 175.30 GiB across six sha256-
verified split parts — on both Radeon AI PRO R9700s, as a ROOT unit (the
frank-large shape: no container, no generated unit).
The deployed contract:
- Placement law: layer split -ts 1,1 with -ngl 999; the routed experts
and the 51B-param n-gram (PLE) table are COMPUTED in host RAM via
-ot exps=CPU,ngram_embeddings=CPU (the x4-pair measurement: streaming
over PCIe loses at 3.7 t/s, computing in RAM does not care).
per_layer_token_embd.weight=CPU is required by the shared MTP head's
borrow path (a GPU-resident borrow crashes the draft load).
mmap stays on; --cache-ram 16384.
- Spec chain draft-mtp,ngram-mod: the shared Q8_0 MTP head
(mtp-...-shared-Q8_0.gguf, 2,786,568,256 B) at depth 2, with the
draftless ngram-mod matcher chained behind it — the only legal way to
layer two spec implementations in one process.
- Vision explicit: --mmproj names the BF16 projector (this build never
autoloads; a server without it comes up healthy and text-only).
- Reasoning: --reasoning-format deepseek → reasoning_content; the
Qwen3.8 template's {xhigh, medium, low} reasoning_effort gate folded
by reasoningEffortMap in models.yml.
- One stream, the model's full 262,144 context; greedy server default
with Qwen-recommended sampling riding compat.extraBody.
The deployer's deploy_llm_primary is rewritten for the root unit: it
gates every artifact at its declared byte count (stat -c %s, never wc -c
— which read the whole 377 GiB lucebox file for minutes on 2026-09-26),
retires the lucebox user quadlet to dated home trash, waits out port
7770, relinks/enables/restarts the root unit and waits up to 1800 s for
/health. The quadlet-era tree under dankserver/frank-primary/ is
archived (dankserver/.AGENTTRASH/20260926-llamacpp-cutover/ on this
box; git history everywhere else) and its planner mapping replaced by
the scripts/ trio.
Observability flips back to the llamacpp namespace: a frank-primary
Prometheus job at 192.168.0.235:7770 (cross-host LAN — the netavark
published-port quirk that forces host.containers.internal for the two
Dankclaw endpoints is same-host only), the LlmEndpointDown runbook
text, and the dashboard description for three llama.cpp endpoints.
Docs: the new runbook (frank-primary-llamacpp-qwen38-flash-next-q8.md),
RETIRED banners on the lucebox and radiance runbooks and their index
entries, the AGENTS.md fleet section rewritten for the root-unit shape,
and the homepage frank-medium card renamed to frank-primary with the
Q8_0 Flash-Next spec (docs.html #models and the modality note updated:
images on frank-primary and frank-small, audio on frank-small only).
models.yml: the dankserver provider is "llamacpp (dankserver)" serving
frank-primary, contextWindow 262144, maxTokens 65536 (compaction-gap
law: 183,501 + 65,536 <= 262,144), input [text, image].
Suite: 346 green (frank_primary_unit.bats rewritten for the root unit —
16 tests pinning the placement law, the shared-head requirement, the
explicit projector, HIP env hygiene (BIND_HOST, never bare HOST), and
the deployer's byte-count gates; the quadlet-era planner and
changed-tests mappings rewritten for the trio; the observability scrape
test asserts three llamacpp jobs again).
Rollback: the lucebox image (localhost/lucebox-rocm:gfx1201) and the
378 GiB payload stay on the host; the Qwen payload stays fetched, so a
re-swap in either direction is a redeploy, not a refetch.
Replaces frank-medium (vllm-radiance TP=2) on :7770. The 377.2 GiB
43-layer MoE (256 routed experts, top-6) is mmapped: dense work and the
MLA KV cache on the R9700, routed experts computed on CPU from the
page cache (246 GiB RAM). The unit bypasses the image entrypoint
(Qwen/DFlash-only flag set) and execs luce_server with the DS4 launch
shape; --ds4-fused-decode's monolithic-load fallback IS the serving
mode here. deploy_llm_primary gates on image + model byte count,
stops/masks the radiance unit, waits for :7770, installs and starts.
Runbook: frank-primary-lucebox-ds4.md.
Each LLM card now carries a details object (role/spec/use) rendered as a
bold pale-green role, a bright-green spec line, and a dim 'Use:' line.
app.js descHTML() falls back to the plain description for non-LLM cards;
new .desc-role/.desc-spec/.desc-use CSS + bare-mode reset.
Roles made neutral (no nicknames): frank-large = deep reasoning,
frank-medium = primary model, frank-small = fast utility model.
The /docs model guide was fixed — it had drifted to retired model info
(frank-large as Qwen3.8-Flash-Next 87.3 GiB/200k, frank-medium as FP8
~31 GiB) — now Qwen3.8-27B UD-Q8_K_XL 31.5 GiB/262k and MXFP4/W4A8
19.4 GiB, reordered large->medium->small. bats 30/30 + live render.
The auth works via the modern /Users/AuthenticateByName endpoint (what the
web UI uses). The 401 was a test artifact from probing the dead legacy route.
No config change needed; LDAP-Auth plugin + authentik 2026.8.3 outpost healthy.
The UserWriteStage writes prompt_data['password'] via user.set_password().
A PasswordStage only authenticates the current password. Document the
correct mechanism (password field on the default-user-settings prompt +
empty-password guard policy) and the throwaway-user verification.
The default-user-settings-flow was missing a PasswordStage, so users could
edit name/email/locale but not their password. Added a PasswordStage
(InbuiltBackend) bound at order 101. Documented the flow's stage table and
the two ORM gotchas (MTI through model, required backends array).
The local username/password path is retired (ENABLE_LOGIN_FORM +
ENABLE_PASSWORD_AUTH false, ui.enable_login_form DB override false) and
qa-e2e-admin is out of the authentik Admins group. Documented in the SSO
section and History.
Records the LDAP-Auth plugin wiring: the Jellyfin LDAP provider +
outpost on Dankclaw, the svc-jellyfin-ldap service account, the
jellyfin_users/jellyfin_admins access groups, and the key gotcha that
the bind credential is a PASSWORD (set via Recovery), not an API/app
token — the default auth flow's standard-password backend rejects
tokens with 'flow error password: Invalid'. Verified end-to-end:
alex (in jellyfin_users) logs in with their authentik password and the
plugin creates the matching Jellyfin user.
Boot 2026-09-22 at 0.98 fill: KV pool auto-sized to 425,047 tokens (1.62x
the 262,144 window) with 14.42 GiB per card available for KV. The DFlash2
drafter on the Swift finetune measures mean acceptance length 7.00 / 85.7%
(easy) and 5.08 / 58.3% (varied) — it survives the base-model change.
Replaces the estimated ~420k / ~17.6 GiB figures with the measured values.
0.99 failed the vLLM startup memory check before weight load: the R9700
reports 31.86 GiB total but only 31.23 GiB free at device-init (the ROCm/HIP
context + NCCL + radiance custom all-reduce consume the 0.63 GiB gap), so
0.99 (31.54 GiB) exceeded free and the container crash-looped. 0.98 (31.22
GiB) is the ceiling and the prior production value.
Also fixed the runbook KV math that had shipped with the 0.99 change: the
~730k pool estimate was impossible (MXFP6's 25.16 GiB weights > MXFP4's
19.37 GiB, so the pool is smaller, not larger than MXFP4's 486k) and the
34.2 GiB/card figure was not the R9700 total. Recomputed ~420k tokens at 0.98
fill.
The serve script now passes --chat-template-file pointing at the
Qwen-Sharp v22.4.0 template. Without it the binary falls back to the
built-in peg-native template, which produces a ~73,600-token prompt for
a ~250-token request. With the sharp template wired in, the same request
produces a 62-175-token prompt.
Knowledge-base follow-up to a45f396. The runbook, both index one-liners, and
the AGENTS.md topology row all still declared FP8 active; flipped to
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq with unquantized bf16 KV (pool
486,464 tokens, full 262,144 window), and recorded the causal nuance: fp8 KV
was on both arms, so the FP8-vs-MXFP4 gap was weight precision, not KV - clean
KV is a quality hypothesis for W4A8, not a throughput one. No verdict yet.
The quadlet's "TEXT ONLY ... 400 here" comment was wrong: vision works on the
MXFP4 checkpoint (Qwen3_5ForConditionalGeneration carries the vision tower
through the Gemma4 merge path, unquantized) - verified by naming a solid red
48x48 PNG "Red". Comment corrected; harness models.yml input stays [text,image].
Docs-only; no re-deploy (the endpoint already serves vision).
The LLM backends section said two ROCmFPX endpoints; the fleet now serves
three (frank-large/medium/small) with different runtimes. Rewrite it, fix the
gotcha that claimed host.containers.internal does not resolve on a bridge
(it does — verified end-to-end for both on-host endpoints), and add a History
entry for the 2026-09-21 model-picker rebuild.
The per-key config DB already pointed web search at SearXNG (engine=searxng,
query URL http://192.168.0.143:8081/search), but only inside the data volume.
The quadlet now carries the same values as seeding env vars, so a fresh data
dir comes up wired, and the doc records the access path: the internal
host-published port, not the search. vhost — the Anubis->Authentik guard
chain is browser-only, and the bridge egress sits in SearXNG's pass_ip, so
the 4-req/IP/hour JSON cap never applies.
The first cutover crash-looped the unit with 'invalid split file name':
the fetch script concatenated the three split parts into a single GGUF,
but part 1's header carries ggml.file_count=3, so the byte-correct
concatenation still advertises itself as a split and the loader demands
a -NNNNN-of-NNNNN filename.
- serve-frank-large.sh: -m points at part 1; llama.cpp loads the split
natively (parts 2-3 auto-discovered in the same directory)
- deploy-remote.sh: LLM_LARGE_MODEL gate path follows
- download-frank-large-unsloth.sh: assembly removed; the parts ARE the
canonical payload, verified against live Hub LFS oids
- docs: runbook model section + troubleshooting note, AGENTS.md, log
- the redundant 87 GiB assembled file is staged on dankclaw at
~/.AGENTTRASH/20260921-frank-large-assembled/
Both pages now group their panels under section headings rendered by shared
renderSection()/renderPanel() components in app.js (SECTION_ORDER fixes each
page's order: user = AI, Web Apps, Media Library Apps; admin = Media Library
Apps, Observability, AI, Infrastructure). The WIP badge splits into WIP: OPEN
and WIP: CLOSED via the wip value ("open"/"closed"); the checker passes the
value through as wip-open/wip-closed. The frank-large card follows the
unsloth llama.cpp cutover, frank-medium follows the FP8 checkpoint, and the
/docs model guide matches. Tautulli (Plex statistics, 192.168.0.235:8181)
joins the admin page's Media Library Apps section with an open link.
Only the 8084 forward is exposed; the desktop GUI runs inside the
container netns and is internal-only. Drop the now-unneeded
CUSTOM_HTTPS_PORT (no host-port conflict exists inside the netns).
lscr.io/linuxserver/calibre:9.15.0 (desktop build) running calibre-server
as the s6 main service on internal 8090 -> host 8084, auth-gated with a
plaintext userdb (mode 600) bootstrapped once by the deploy handler.
The desktop GUI is kept on 8080/8182, basic-authed with the same
credential pair via the machine-local env file.
Frank
