serverAppConfigs/serverAppConfigs-docs/LLM/index.md

LLM Fleet — Knowledge Base

Dankserver runs the radiance engine (StillDeadcode) on frank-primary (digest-pinned stilldeadcode/radiance image, engine as entrypoint, driving podman run directly, root systemd unit, :7770): the OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint at NATIVE block-FP8 precision (converted no-recipe to qwen4exp-fp8-native.rad — see the runbook for the descriptor-staging law), on BOTH R9700s TP=2 (--tp-wire wht6), experts in a 40 GiB/rank pinned host pool + VRAM cache (ratio 0.82), the 47.7 GiB PLE table pinned to host RAM, fp8 KV, MTP spec decode depth 3, 2 × 200,000 first boot. Rollback tags frank-primary-freetoken-pr587-20261005 and frank-primary-vllm-shapeG-20261003), and Dankclaw runs two endpoints — halogen container (frank-large, port 7770: ghcr.io/peonist-ai/halogen @ pinned digest serving peonist-ai /halogen-qwen3.8-27b, a 33.4 GiB .hgn checkpoint that loads in no other engine, DFlash2 spec decode, ONE slot at 200,700 ctx, text-only — vision lives on frank-small) and llama.cpp CUDA (frank-small, port 7771: ukisai Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S-mtp GGUF — 27B with the MTP head IN the file, ONE 131,072 conversation at Frank's 128k cap, q8_0 K + q8_0 V (FP8 class), vision via the base-model projector, on the RTX 3090).

2026-08-22: the former vllmScripts repository is merged into the serverAppConfigs repository. This knowledge base moved from the old repo's serverAppConfigs-docs/ root to serverAppConfigs-docs/LLM/. The fleet scripts and units now live in the repo's scripts/ directory.

Deployment Topology

graph LR
    subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
        A[2x Radeon AI PRO R9700
gfx1201, 32 GiB each]
        B2[radiance StillDeadcode, digest-pinned image
podman run, frank-primary, port 7770]
        C[Qwen3.8-Flash-Next FP8 OFFICIAL, native block-FP8
2x R9700 TP=2 wht6, experts 40 GiB/rank pinned pool + VRAM,
PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 3,
2 x 200000, froggeric v22.5]
    end

    subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
        E2[RTX 3090 (PCIe x4 dock) +
gfx1151 iGPU 114 GiB GTT]
        E[halogen container gfx1151, digest-pinned image
frank-large, port 7770]
        F[halogen-qwen3.8-27b .hgn p1w4d-d2 + DFlash2 drafter
1 x 200700 ctx, d2 KV, text-only]
        G[llama.cpp master CUDA sm_86
frank-small, port 7771]
        H[Swift-1.5-27B GSQ-RCO IQ3_S-mtp GGUF
1 x 131072 ctx, q8_0 K + q8_0 V, draft-mtp depth 3, mmproj vision]
        E2 --> E --> F
        E2 --> G --> H
    end

    CLI[LLM Clients / API] -->|HTTP :7770| B2
    CLI -->|HTTP :7770| E
    CLI -->|HTTP :7771| G

Servers

Server	GPU	VRAM / Memory	OS
Dankserver System Specs	2x AMD Radeon AI PRO R9700	2x 32 GB VRAM, 246 GB RAM	Ubuntu 26.04
Dankclaw System Specs	AMD Radeon 8060S (Strix Halo)	gfx1151 unified memory	Ubuntu 26.04

Active Deployment

• frank-large-halogen27b-dflash2 — ACTIVE since 2026-10-05 (root systemd unit + podman, port 7770): the digest-pinned ghcr.io/peonist-ai/halogen image serving peonist-ai/halogen-qwen3.8-27b (qwen3.8-27b-p1w4d-d2.hgn, 33.4 GiB) with the DFlash2 drafter on ONE slot of 200,700 ctx (slots>1 disables speculation — engine law). Publisher card 31.7 t/s mean across eight text workloads. Accepted tradeoffs: TEXT-ONLY (the engine ignores the vision encoder; frank-small owns images) and the CENSORED base model (the ROCmFPX AGENT file was Uncensored by construction). No server-side sampler — the thinking recipe rides the wire in models.yml. Rollback: revert the swap commit; the ROCmFPX payload + fork build + build-rocmfpx.sh pin all remain on disk
• frank-small-llama-swift15-iq3s-mtp-131072 — ACTIVE since 2026-10-05 (evening) (root systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF IQ3_S-mtp (27B, the checkpoint's own MTP head IN the file — --spec-type draft-mtp, depth 3) on the RTX 3090, ONE 131,072-token conversation at Frank's 128k cap (a second 131,072 slot misses the 2-3 GiB floor) with q8_0 K + q8_0 V (FP8 class, 34 KiB/token vs 64 at f16) and vision via unsloth's mmproj-Qwen3.8-27B-F16.gguf — no Swift projector ships, and the base vision stack was hash-proven byte-identical to Swift's (6/6 tensors). The GSQ-RCO "IQ3_S" tier is a per-tensor allocation mix of stock types. Replaces BOTH same-day payloads: the live MiMo Q6_K shape (below) and the Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never deployed
• frank-small-llama-mimo-q6k-2x200704 — RETIRED 2026-10-05 (evening; rollback target; payload on disk): llama.cpp master (CUDA sm_86) serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q6_K, TWO slots of 200,704 with FULLY UNQUANTIZED KV (f16 K + f16 V) and the mmproj for vision — 94.4 tok/s decode (the rung costs a measured -10 % vs Q5_K_M: weight-bandwidth-bound), batch 4096 / ubatch 2048 since 2026-10-05 (+4.6 % prefill, measured), 2,279 MiB of the card left free. This was the LIVE payload when Swift cut over. The ubatch A/B table and the probe-size trap (>=256 px images or the model says "video") live inside
• frank-small-llama-mimo-q5km — RETIRED 2026-10-02 → 10-03 (rollback target; payload on disk; owns the measured boot-pair table): the same runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower — 104.8 tok/s, 3,571 MiB true free
• frank-primary-radiance-qwen38-native-fp8 — ACTIVE since 2026-10-05 (root systemd unit, port 7770): the StillDeadcode radiance engine (C++/HIP, libr4d gfx1201 kernels, digest-pinned 287 MiB image, engine as the image entrypoint) serving the OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint at NATIVE precision — converted no-recipe to qwen4exp-fp8-native.rad (~161 GiB) after dropping quantization_config in a staging dir (the plugin descriptor-selection law: nothing claims (qwen4exp, "fp8"), the shipped plugin claims (qwen4exp, ""); per-weight encodings are data, not the descriptor) — on BOTH R9700s TP=2 (--tp-wire wht6 fused all-reduce), experts in a 40 GiB/rank pinned host pool + VRAM cache (ratio 0.82 = the measured 99 %-hit point), the 47.7 GiB PLE n-gram table PINNED to host RAM (--ngram-placement ram), fp8 KV 7.32 KiB/token, MTP spec decode depth 3, 2 × 200,000 first boot, froggeric v22.5 template, vision + tools + reasoning_content auto, /metrics scraped. Rollback: git tag frank-primary-freetoken-pr587-20261005 (FreeToken image + orcarouter payload on disk); older rollback frank-primary-vllm-shapeG-20261003
• frank-primary-freetoken-pr587 — RETIRED 2026-10-05 (same day; rollback target): FreeToken at PR #587 head 77a7201 (branch rocm714_torch214) built on the host into the local image frank-primary-freetoken:pr587-77a7201 (Dockerfile: digest-pinned rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev-symlink law, curated --no-deps install per docs/install_amd.md), serving the SAME orcarouter FP8 payload on ONE R9700 — experts in pinned host RAM (~120 GiB + 47.7 GiB PLE), GPU LRU at 0.10, KV pool 299,968 tokens (multiple-of-64 law), 2 slots, parsers AUTO (thinking → reasoning_content). Measured: boot ~2.3 min, decode 17.1 tok/s single-stream, vision "Red" probe passes, native tool_calls; TP=2 does not boot on ROCm (pynccl JIT) — second card idle. Superseded by the radiance swap for TP=2; the image + payload stay on disk until the rollback window closes
• dankserver-qwen38-fp8-expert-offload-tuning — RETIRED 2026-10-05 (rollback target; the vLLM trio + env lore live in the tag): the tcclaviger RDNA4 vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both R9700s, TP=2 with automatic expert offload — --max-model-len 200704 x 3 slots, fp8 KV, pool 3,150,000,000 B/rank (measured 398,658 tokens = 1.99x), MNBT 1024, gather 32x64, no spec decode, prefetch off, plus --expert-precise-memory (arena 17.22 GiB / 7,522 slots), GPU_MAX_HW_QUEUES=2 and a mounted compile cache. Why --max-num-seqs must stay 3 (SEQS 2 cannot compile this model), why the arena's levers are max_model_len and the overhead term rather than the KV pool, and the measured envelope
• frank-primary-glm53-flash-w4a16 — RETIRED 2026-09-30 → 10-01 (the failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro): the tcclaviger RDNA4 vLLM fork (digest-pinned tcclaviger/vllm@sha256:da4029666…, plain podman run, no shim) serving Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric int4 group 128, ~170 GiB, 34 shards + 2 extras, 113,074 tensors — on BOTH R9700s, TP=2, with the MoE experts offloaded to pinned host RAM (12,096 experts/rank, 73.64 GiB) behind a 14.51 GiB / 2,383-slot VRAM LRU arena; the MoE kernel is libr4d's grouped W4A16 GEMM (R4dW4a16MoEMethod, reached because the checkpoint's quant_method: auto-round is overridden to inc — a compressed-tensors W4A16 checkpoint of the same model has NO offload recipe and refuses to serve). 131,072 ctx x 2 seqs, bf16 KV, util 0.98, prefix caching, vision and glm47 tool/reasoning parsers, MTP OFF and prefetch OFF by measurement. Measured decode 9.9-10.0 tok/s, prefill 918 tok/s — the 35 tok/s target is NOT met, bounded by the fork's expert-gather read path (~8.2 GB/s effective vs a 22.74 GB/s link) at a 0.53-0.66 cache hit rate. Rollback: commit 2339fae (the qwen FP8 + MTP4 shape of 2026-09-29 → 09-30)
• frank-primary-r9v-radiance-qwen38-q8 — RETIRED 2026-09-30 (two swaps back; payload and R9V state dirs intact on disk, git tag frank-primary-r9v-q8-20260929 is the last pre-cutover commit): frank-primary-r9v-radiance-qwen38-uncensored — RETIRED 2026-09-27 (same-day swap to the Q8_0 payload; rollback target, state + payload intact on disk): the R9V radiance-vLLM fork serving Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) + blockfp8 MTP4 (depth 4) + F16 vision on BOTH R9700s — TP=2 tiered experts (UVA pinned host copy, 62 hot + 160-slot LRU on rank 0, 428 static on rank 1), PLE n-gram on ssd-mmap, CED on, 196608 ctx. Cut over from llama.cpp the same day; rollback target frank-primary-llamacpp-qwen38-flash-next-q8
• frank-primary-llamacpp-qwen38-flash-next-q8 — RETIRED 2026-09-27 (rollback target; the llama.cpp b11030 endpoint of 2026-09-26): unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X, bundled runtime) serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 across both R9700s — routed experts and the 51B-param n-gram table computed in host RAM (-ot exps=CPU,per_layer_token_embd.weight=CPU), the shared Q8_0 MTP head (depth 2) chained behind ngram-mod, BF16 vision, one full 262,144-token conversation, reasoning_content, the placement law, the cutover and rollback
• frank-primary-lucebox-ds4 — RETIRED 2026-09-26 (same-day swap, rollback target; active for hours): the Lucebox DS4.1 engine (localhost/lucebox-rocm:gfx1201) serving Lucebox/DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB) with streamed experts — the fleet's only non-llama.cpp runtime detour
• dankserver-frontier-moe-feasibility — VERIFIED 2026-09-28: which of DeepSeek-V4.1-Flash (552B), MiMo-V2.6-Flash-MOPD (309B) and GLM-5.3-Flash (320B) can run on the 2x R9700 box, on what runtime, and why V4.1 does not fit (475 GiB payload, 189 GiB of it Engram; decode is PCIe-expert-stream bound, ~23 tok/s ceiling at today's residency). Includes the Affinity V4-Flash-to-V4.1 port assessment — reusable expert tiering, four architecture gaps, one of them a silent-wrongness trap on the Engram key
• dankserver-qwen38-fp8-expert-offload-tuning — VERIFIED 2026-09-29: the flag-level decode-tuning campaign on the ad hoc qwen-fp8 container (tcclaviger RDNA4 fork, orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8, TP=2 across both R9700s). >35 tok/s MET: 37.44-40.57 tok/s with --expert-prefetch DELETED — telemetry measured the prefetcher at 36% precision, wasting ~half the gather traffic (the LRU itself runs at 82.45% hit). Prefetch-on incumbent 26.8 survived every other challenger (geometry, spec depth, TunableOp-corrupts, batch knobs, concurrency, ngram-fork-bug); the residual gap to ~60 tok/s is the fork's gather read path (6.5-9.5 GB/s effective vs 22.7 at the same slab size)
• frank-medium-radiance-mxfp4 — RETIRED 2026-09-26 (replaced by frank-primary-lucebox-ds4, then frank-primary-llamacpp-qwen38-flash-next-q8): vllm-radiance 0.9.3 serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark W4A8) with the z-lab DFlash2-FP8 drafter as ONE tensor-parallel process (TP=2) behind :7770, on unquantized bf16 KV (KV_DTYPE=auto, cut 2026-09-21). The FP8 checkpoint is the on-disk rollback. The three-arm battery that chose TP=2, the auto-sized KV pool, and the verified capability battery (tools, reasoning)
• frank-medium-model-swap — how to swap the checkpoint behind frank-medium: the streamlining toolchain (hf-manifest.py, hf-fetch.sh, swap-frank-medium.sh), the quantization-aware entrypoint contract, the quadlet edit, the deploy, the capability battery, and the rollback path. The 2026-09-20 MXFP4 → FP8 swap is the worked example
• frank-medium-llamacpp-dflash2 — RETIRED 2026-09-16 → 2026-09-20: mainline llama.cpp serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the DFlash2 Q4_K_M drafter as ONE layer-split process (32:33 over two 163,840-token slots); superseded by the radiance TP=2 cutover, and the rollback target. Keeps the batch/ubatch optimum curve and the per-layer split arithmetic
• Dual-R9700-Cooperation — whether the two R9700s can serve ONE model: the RCCL/PCIe-atomics wall on the old TUF GAMING B650-E, the 2026-09-20 ProArt B850-Creator x8/x8 swap that cleared it, and the measured three-way verdict (radiance TP=2 95.8 t/s decode vs layer split 45.7 vs llama tensor mode 30.5) that put TP=2 in production
• frank-medium-rocmfpx — PRIOR frank-medium deployment (llama.cpp era, 2026-09-15): charlie12345/ROCmFPX serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the runtime it replaced
• freetoken-frank-small — RETIRED 2026-09-15, when the RTX 3090 moved to frank-large: the FreeToken v0.1.2 frank-small runbook (batsclamp Huihui-Qwen3.6-35B-A3B-abliterated-FP8), kept as the retirement record. The NAME was reused on 2026-09-16 by a different service on Dankclaw — see frank-small-qwen38-gsq-rco
• freetoken-benchmark-sweep — KV pool vs decode speed sweep (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
• deployment — deploy and update process for both servers
• frank-redo-2026-08-17 — the 2026-08-17 topology change
• frank-large-rocmfpx-q6agent-uncensored-mtp — RETIRED 2026-10-03 → 10-05 (rollback target; payload, fork build and build script on disk): the pinned ROCmFPX fork serving the uncensored protected-heads Q6_0_ROCMFPX_AGENT rung with the repo's MTP-Q4_0 head drafting 4 tokens, one 262,144 window, f16 KV; measured 27.6-28.1 t/s prose decode at 86.8 % acceptance. Replaced by the halogen DFlash2 endpoint; the ngram-mod-onto-MTP rejection (-15 %, recurrent-state rollback) stays a live lesson for this architecture.
• frank-large-rocmfpx-q8agent-mtp — RETIRED 2026-10-02 → 10-03 (the rollback target; payload still on disk): the same pinned ROCmFPX fork serving the ALIGNED Q8_0_ROCMFPX_AGENT (ftype 115, 17.0 t/s, MTP acceptance 0.462) — the MTP-vs-none +123 % A/B, the trap catalogue
• frank-large-gufo-qwen38-27b — RETIRED 2026-10-02 (history; the payload dirs were removed from disk 2026-10-03, so this rollback is a re-download) (root systemd unit, port 7770): gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw, serving unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL with the z-lab DFlash2 Q4_K_M drafter — ONE session at the full native 262,144 window, unquantized f16 KV. Rolled back by Frank after one day of frank-large-vllm-mimo-v26-bf16. The two from-source build workarounds, the client-contract delta, the measured prefill/decode trade against llama.cpp
• frank-large-vllm-mimo-v26-bf16 — RETIRED 2026-10-01 (one day; rollback by Frank after the autoresearch loop): the digest-pinned kyuz0/vllm-therock-gfx1151 image serving XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16, 4 x 262144 ctx, CROSS-MODEL MTP from Qwen/Qwen3.5-9B — kept for the measured record: battery decode 6.19 t/s (MTP-2) vs 9.01 t/s (no draft — spec NET- NEGATIVE on real chat+reasoning traffic), the roof math, the greedy-text numerics trap, and the death of every ≥6-bit vLLM quant path on gfx1151 (quark int8 rejected, FP8 kernels CDNA-only, torchao unwired)
• frank-large-unsloth-qwen38-27b-q8xl — RETIRED 2026-09-28 (replaced by frank-large-gufo-qwen38-27b; rollback target): stock llama.cpp (unsloth b11030, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter (depth 4) on the gfx1151 iGPU — one full 262,144-token conversation, q8_0 KV (the build lacks fp8), text-only, Qwen-Sharp v22.4.0 template, the 114 GiB GTT memory model, client contract, troubleshooting
• frank-large-kairic-edge-qwen38-27b — RETIRED 2026-09-21 (same-day swap, replaced by frank-large-unsloth-qwen38-27b-q8xl; rollback target): the Kairic Edge runtime (ciru-ai/ROCmFPX v1.2, built from source) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (IU4 W4A4, 27B dense) on the gfx1151 iGPU — one full 262,144-token conversation, MTP depth 4 + n-gram drafting, the three PromptForge sidecars, compatibility mode, the 114 GiB GTT memory model, client contract, troubleshooting
• frank-large-unsloth-qwen38-flash-next — RETIRED 2026-09-21 (same-day swap, replaced by frank-large-kairic-edge-qwen38-27b; rollback target): the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (125B MoE) on the gfx1151 iGPU — one 200,000-token conversation, MTP depth 2 via the explicit -md shared-Q8_0 head, the 114 GiB GTT memory model, client contract, troubleshooting
• frank-large-ciru-orca — RETIRED 2026-09-21 (replaced by frank-large-unsloth-qwen38-flash-next; rollback target): the CIRU v4.4.1 runtime (private host build, QSA hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU — one 262,144-token conversation, MTP depth 4, 4096 MiB PLE cache, runtime-root gate, client contract, troubleshooting (2026-09-19 → 09-21)
• halogen-flash-server — RETIRED 2026-09-19 (replaced by frank-large-ciru-orca; rollback target): peonist-ai halogen-flash-server 0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one 400,384-position pool, deployed flags, memory model, client contract, troubleshooting (2026-09-16 → 09-19)
• frank-small-llama-mimo-q6k — RETIRED 2026-10-01 → 10-02 (the same rung returned ACTIVE 2026-10-03 at the 2 x 200,704 f16-KV shape — frank-small-llama-mimo-q6k-2x200704): llama.cpp (master, CUDA sm_86 build) serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q6_K on the RTX 3090 — TWO native 262,144-token sessions with ASYMMETRIC KV (K f16 unquantized + V q8_0, which forces flash-attn), ~20.9/24.6 GiB measured, text-only (the GGUF has no vision), reasoning_content, no spec decode; payload still at /home/dank/models/MiMo-Q6K/
• frank-small-sglang-mimo-v26 — RETIRED 2026-10-01 (hours; replaced by frank-small-llama-mimo-q6k for the asymmetric KV SGLang cannot express; venv still on disk, INT8 payload staged to ~dank/.AGENTTRASH/20261002-retired-llm-payloads/): SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6 INT8 W8A16 AutoRound, 2 x 262144 ctx, fp8 KV, in-checkpoint vision, 83.9 tok/s decode
• frank-small-gemma4-12b — RETIRED 2026-09-17 → 2026-10-01 (replaced by frank-small-sglang-mimo-v26 → the llama.cpp MiMo shape; payload staged to ~dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/): mainline llama.cpp serving HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX 3090, with the separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window into ~12 GiB.
• frank-small-vllm-gemma4 — RETIRED 2026-09-19 (same-day revert, rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16 Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647- token pool, 2 concurrent full-context sessions, native text+image+audio.
• frank-small-qwen38-gsq-rco — RETIRED 2026-09-17 (rollback target): the prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with the model's own in-file MTP head and f16 KV at 131k
• x4-pair-experiment — MEASURED 2026-09-16: what mainline llama.cpp can and cannot do with the 3090 + Strix Halo pair, over their only link (the 3090's Gen4 x4). The tensor inventory that decides any split, the single flag (-ot) that kills per-tensor placement across devices, the two-device layer splits that work (+7 % decode at 64k, cross-vendor included), the harness-parsing trap that invalidated the first round, and the iGPU-only result that matches halogen's deployed decode rate — at a memory cost the co-tenants cannot pay
• frank-large-ik-uncensored — RETIRED 2026-09-16 (rollback target): the ik_llama.cpp runbook (root systemd unit, 2026-09-15 → 09-16) serving mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090 (dense/attention/KV) + LPDDR (experts/PLE) hybrid; its measured envelope and decode root-cause chain are the record of why the fleet went back to halogen
• dankclaw-iommu-translation-off — why Dankclaw boots with amd_iommu=off since 2026-09-11 (13-16 % prefill, measured upstream), what it cost (the NPU endpoint), what was deliberately not copied from the reference machine, and how to revert
• kingjones-strix-rocmfp4 — PRIOR frank-large runbook and rollback target: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params (56B MoE + 51.2B PLE), single 113.47 GiB GGUF, on the card's pinned kingjones30/ROCmFPX fork (2026-08-30 → 2026-09-11)
• ubatch-benchmark-large — active-model llama-bench ubatch sweep plan and results
• frank-medium-batch-ubatch — batch/ubatch sweep on the 2x R9700 layer split (2026-09-16): the ubatch optimum at 512, why the logical batch is free, and the split rebalance that followed it
• kingjones-q8agent-rocmfpx — the retired Q8_0-AGENT investigation and set_rows failure
• eemin-rocmi4-mtp — the former caf-eemin Qwen3.8-27B embedded-MTP model on Dankclaw (retired)
• dealign-ornith-crack — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (frank-small, retired 2026-09-02)
• ikllama-ornith-dankserver — the retired ik_llama.cpp runtime for frank-small (2026-08-26): q8_0 + Hadamard KV cache, split-mode none (single-GPU), mandatory FA_ALL_QUANTS build flags
• kingjones-strix-mtp — the former Q6 AGENT runbook (restored 2026-09-01 after Q8_0 failure, then superseded by the STRIX deployment on 2026-08-27)
• ciru-activefpx-promptforge — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
• kairic-edge — the retired Kairic Edge IU4 experiment (deployed and reverted 2026-08-22)
• Dankserver System Specs — the 2x R9700 box (frank-primary)
• Dankclaw System Specs — the Strix Halo gfx1151 box (frank-large on the unsloth b11030 runtime since 2026-09-21, frank-small on the RTX 3090)
• frank-npu — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as the retirement record — the IOMMU/SVA err=-19 root cause and where every dependent route moved

CIRU Reference (Dankclaw)

• ciru-activefpx-promptforge — the RETIRED v3 model, ROCm 7.15 toolchain, and build history (on Dankclaw; the slot now serves the kingjones ROCmFP4 model, see kingjones-strix-mtp)
• kairic-edge — Kairic Edge IU4 runtime: build runbook, binary identity, and experiment summary (retired 2026-08-22)

DFlash2 Reference (archived)

• dflash2-large-vulkan — the retired DFlash2 Vulkan deployment (2026-08-21)

vLLM Reference (historical)

• VLLM Overview — installation, CLI, API, architecture, quantization, parallelism
• VLLM Index — quick reference with commands, endpoints, and parameter table
• Running Qwen3.6 35B A3B FP8 on Dankserver — Qwen3.6-35B-A3B hardware notes and sizing (this architecture family is what the retired frank-small served in the Huihui abliterated FP8 variant — see freetoken-frank-small; the Ornith-1.5-35B-A3B CRACK checkpoint that preceded it is archived at dealign-ornith-crack)

Memory Management

• RAM CPU Offloading in VLLM — KV cache swap between GPU VRAM and system RAM
• Hybrid CPU GPU Processing Dankserver — NVIDIA GPU compute with system RAM offload
• Shared Memory Architecture Dankclaw — AMD unified memory and ROCm configuration

Retired Topology (2026-08-17 redo)

• Qwen3.8 PP Deployment — the retired pipeline-parallel head + worker
• qwen3.8-spec-matrix — Qwen3.8-27B spec history (superseded by FastMTP)
• FastFlowLM NPU Inference — the retired FastFlowLM NPU endpoint
• frank-large-benchmark — backend/spec benchmark behind the old head
• dankserver-vram-growth — VRAM/RAM growth behind the retired cuBLAS crash
• DFlash DSpark Speculative Decoding — retired DFlash/DSpark drafting
• Quantization by Hardware Architecture — which quant format fits which GPU/CPU

Pipeline Parallelism (archive)

• Pipeline Parallelism/index — index of multi-node inference research. The PP pair is retired. These pages are research history.

Runtime Research (llama.cpp family)

• Llama Cpp Runtimes — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: positioning, quantization, server fit
• Llama Cpp Code Diff — code-level quantization diff of the three runtimes + server recommendations
• SGLang — the structured-generation serving engine; when it fits Dankserver (2x R9700) vs Dankclaw (Strix Halo)

Cross-Cutting References (skill distillation, 2026-10-05)

One polished reference page per skill-cluster, distilled from the managed skill library into the vault (see log for the wave plan):

• GGUF Compatibility — what a GGUF's tensor histogram and header demand of a loader; how the runtime is chosen by the file
• Llama Cpp Serving Sizing — slot/pool/KV sizing laws, boot-shape laddering, llama-bench discipline
• Llama Cpp Multi-Device Splits — layer vs tensor splits, -ts/-ot laws, drafter placement
• PCIe Link and Atomics — the link walls, atomics routing, and the heterogeneous-pair verdicts
• RDNA4 Container Serving — bring-up recipes for RDNA4 containers
• RDNA4 MoE Expert Offload VRAM Budget — the plan law, every arena lever, and why free VRAM is not arena
• vLLM Boot Forensics and Config Interrogation — reading a boot that died, and interrogating a running fork's config
• vLLM Porting and Compile Triage — porting checkpoints across vLLM versions/forks; torch.compile/dynamo breakage triage
• LLM Bench Windows and Measurement — window discipline, honest A/B, the window script laws
• LLM Performance Diagnosis — slow prefill/decode triage on live endpoints, IBS attribution, prefix-cache routing
• Model Swap and Rollback — the swap procedure, payload preflight, research verification, second-endpoint bring-up
• Tcclaviger Fork Ops — the fork's sweep harness, measured knob verdicts, large-MoE recipe, compile-cache persistence
• Heterogeneous Split Triage — triage for unequal-accelerator split proposals (the no-go verdict format)
• FreeToken Builds — the CUDA and ROCm build paths with their dependency traps

These sit alongside the operational pages under ../Operations/index.md and ../Agent/index.md.

Related

• Project knowledge base — the Podman app stacks on the same two hosts. OpenWebUI (Dankclaw) consumes the on-host frank-large endpoint at :7770. The Gitea instance on Dankclaw hosts this repository.
• Deployment Flow — the service-scoped git-based deploy with inference isolation