dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
fix/zealot-2x262144-f16-kv
serverAppConfigs/serverAppConfigs-docs/index.md
T
Frank 71bc1735d3 docs(frank-zealot): re-key the exact-KV shape across the vault 
AGENTS.md: topology row, the frank-zealot runbook section (shape, KV
accounting, MTP verdict, compaction-hub concurrency), and two Key
Lessons bullets -- the draft-context bill restated with the real knob
names, plus a new lesson that a quant's scale overhead belongs in its
byte rate (the 6.25 % invented by doubling 13,056.00 MiB).

LLM/frank-zealot-llama-prism-bonsai9-mtp.md: retitled to the live shape;
the 3-slot q8_0 boot ledger and its measurements are relabelled as the
arm this replaced (byte-identical numbers, honest attribution), the
corrected f16 rate derivation is added, the MTP census is labelled
q8_0-era, the fallback rungs are re-derived for the 524,288 pool, and the
boot-ledger section carries an explicit PROJECTION that must be replaced
with the measured ledger from the first f16 boot.

Pointers re-pointed (live-shape sentences only; every historical ledger
left byte-identical): LLM/index.md, vault index.md, Dankclaw/OpenWebUI.md,
and the retired-runbook "today" pointers in mimo-q5km, mimo-q6k,
qwen35-9b-q6k, swift15, sglang-mimo-v26. log.md gains the dated entry.
2026-10-18 21:02:42 -07:00

34 KiB
Raw Permalink Blame History

Server App Configs

Podman deployment configuration for two homelab servers: Dankserver and Dankclaw.

This repository is the single source of truth for both hosts. It tracks the container stacks and the LLM inference fleet. Edits flow through this repo, then deploy to the targets with scripts/deploy.sh (one script per target covers both workloads).

2026-08-22: the former vllmScripts repository (LLM fleet) is merged into this repo with its full git history. Its knowledge base now lives at serverAppConfigs-docs/LLM/index.md.

Hosts

Host IP Role OS
Dankserver 192.168.0.235 Media & library stack Ubuntu 24.04
Dankclaw 192.168.0.143 AI, search & development stack Ubuntu 26.04

Stacks

  • Dankserver Stack — RomM, Copyparty, Calibre, Calibre-Web, Caddy
  • Dankclaw Stack — SearXNG (via Gluetun), ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Caddy

Architecture

Dankserver Storage

Dankserver Services

  • RomM — retro game manager
  • RomM-Scraping — metadata providers, cover art, naming conventions
  • Copyparty — file sharing and streaming
  • Jellyfin — media server (movies, TV, music)
  • Calibre — e-book library engine
  • Calibre-Web — e-book web frontend
  • Plex — media server on Dankserver (lscr.io/linuxserver/plex container via quadlet; CPU decode only — no GPU passthrough)
  • Tautulli — Plex statistics and notifications (v2.18.1-ls245, quadlet unit)
  • mariadb — RomM database backend
  • redis — RomM cache/queue backend

FrankBook-Pro Services

The *arr stack and qBittorrent run on FrankBook-Pro (192.168.0.112, the macOS mini) as quadlets inside its podman machine — not on Dankserver.

Dankclaw Services

  • SearXNG — privacy metasearch (VPN-routed)
  • SearXNG Providers — unblocking Google & other upstream engines behind the VPN
  • Gluetun — the VPN tunnel container
  • Valkey — SearXNG cache and limiter store
  • ComfyUI — node-based AI image generation (ROCm GPU, gfx1151 build)
  • ComfyUI Qwen-Image Workflows — the two proven Qwen-Image 2.1 workflows + what to add next (LoRAs, Edit, ControlNet, Wan video)
  • OpenWebUI — chat UI for local LLMs
  • Hermes — the Nous Research agent
  • Hermes-Dashboard — the Hermes web management UI (config, keys, sessions)
  • Hermes-WebUI — the Hermex server (loopback API for the iPhone app, SSH-tunneled)
  • Gitea — Git hosting (migrated)
  • Postgres — Gitea database backend
  • Seerr — unified *arr request manager (guard-chained; fronts FrankBook-Pro's stack)
  • Marreta — content-cleaning proxy (guard-chained; shares SearXNG's Gluetun egress)
  • Open Notebook — podcast generator (scripts by frank-zealot, voices by frank-tts); repo-owned config, not deployed
  • Frank TTS Engine — the arm A TTS engine (qwentts.cpp resident tts-server, 1.7 B talker) behind frank-tts on a private bridge; deployed on Dankclaw, CPU-only, publishes no port by design
  • Frank TTS — the 7772 cloned-voice endpoint consumers address: voice naming, pack layout, why the upstream is a container name; deployed, and it is the harness speech role
  • Voice Packs — how a frank-tts voice is built: reference.wav + the pinned engine's speaker artifacts, cut past the 10-second source-material gate

Infrastructure

  • LAN-DNS — name resolution, Pi-hole, router DNS hand-off
  • Pi-hole — the LAN DNS server (quadlet, records, web UI)
  • Caddy — reverse proxy and TLS
  • Anubis — bot challenge proxy fronting the guarded vhosts (chain, policy, gotchas)
  • Authentik — the homelab's SSO / OIDC provider (forward_auth)
  • authentik-postgres — Authentik database service
  • authentik-server — Authentik server + Caddy outpost service
  • authentik-worker — Authentik background worker service
  • Homepage — password-protected service index
  • Homepage Design Grilling — homepage decision log and open questions
  • Observability — Grafana, Prometheus, Loki, Alloy, node-exporter, Alertmanager: the stack, its quadlet deployment, its exposure, and what is still missing
  • Grafana — dashboards (datasource and the LLM fleet dashboard provisioned as code since 2026-09-16)
  • Prometheus — metrics collection (5 alert rules + 1 recording rule, delivered via Alertmanager since 2026-09-17)
  • Alertmanager — alert grouping and Signal delivery (webhook → host-side relay → signal-cli)
  • Node-Exporter — host metrics
  • Loki — log storage
  • Alloy — log shipping (replaces Promtail)
  • FrankBook-Pro — Mac mini *arr + qBittorrent inside podman machine (gluetun-gated)

Research

Skill Distillation (2026-10-05)

The managed skill library distilled into the vault, one reference page per skill-cluster. See log.md for the wave plan and results.

Operations:

Architecture:

Infrastructure:

LLM cross-cutting (full index: LLM/index.md):

Agent:

Desktop / Research:

LLM Fleet

The LLM inference fleet on the two hosts. Full index: serverAppConfigs-docs/LLM/index.md.

Deployment and Topology

  • deployment — unified deploy and update process for both servers
  • frank-archon-radiance-qwen38-iq4r — ACTIVE frank-archon runbook (2026-10-15): the PUBLISHER's container of the official Qwen3.8-Flash-Next (qwen3.8-next-flash-fp8-iq4r-moe.rad, 116,322 MiB, w4nl 4-bit-codebook experts + int8 trunk, Hub-sha256-verified at fetch) swapped in as a MODEL_FILE-only change — the 2026-10-13 shape (slab 25600 / fp8-KV 5120, 3 × 200,000, host pool 44,928, MTP depth 2) carries verbatim; rollback = revert the swap commit (the w4dn8 .rad stays on disk; the fp8-native .rad was deleted 2026-10-15, rebuildable)
  • frank-archon-radiance-swift15-w4dn8 — RETIRED 2026-10-08 → 10-15 frank-archon runbook: Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 .rad (radiance 1.0.13), gated on KLD ≤ 0.03 vs the BF16 container and decode ≥ 30.8 t/s; activates by flipping MODEL_FILE in scripts/frank-archon.env, rollback = revert that line (the FP8-native .rad stays on disk)
  • frank-archon-radiance-qwen38-native-fp8 — the RETIRED 2026-10-05 → 10-08 frank-archon runbook (2026-10-05): the StillDeadcode radiance engine (digest-pinned 287 MiB image, libr4d gfx1201 kernels) serving the OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 at native block-FP8 (no-recipe convert after the descriptor-staging drop) on BOTH R9700s TP=2 wht6, experts in a 40 GiB/rank pinned host pool, PLE 47.7 GiB pinned to RAM, fp8 KV 7.32 KiB/token, MTP depth 3, 2 × 200,000 first boot; rollback is the FreeToken one-card era (tag frank-archon-freetoken-pr587-20261005), then the tcclaviger shape-G era (tag frank-archon-vllm-shapeG-20261003)
  • swift15-campaign-worklog — ACTIVE campaign log: the Swift1.5 → w4dn8 radiance convert and frank-archon swap in flight (branch swift15-radiance-w4dn8); the cutover runbook replaces it at completion.
  • frank-archon-freetoken-pr587 — RETIRED 2026-10-05 (superseded the same day by the radiance cutover above; kept as the rollback runbook): FreeToken built at PR #587 head 77a7201 on the host into frank-archon-freetoken:pr587-77a7201, ONE R9700 with experts in pinned host RAM behind a 0.10 GPU LRU, 2 slots, parsers AUTO
  • frank-redo-2026-08-17 — the 2026-08-17 topology change
  • frank-archon-r9v-radiance-qwen38-q8 — RETIRED 2026-09-30 (two swaps back; rollback target): the R9V radiance-vLLM fork (docker-pinned bundle 2dac17a2, docker→podman shim) serving the unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 shards) + blockfp8 MTP4 + BF16 vision on Dankserver's both R9700s — TP=2 tiered experts (80-slot LRU cache), PLE pinned 54.4 GB in host RAM, CED off, 131,072 ctx single stream, Qwen-Sharp template, root unit with LimitMEMLOCK, the on-host qualified state, the cutover, and the rollback
  • frank-archon-r9v-radiance-qwen38-uncensored — RETIRED 2026-09-27 (rollback target): the R9V radiance-vLLM fork serving the abliterated Dyluhn Qwen3.8-Flash-Next-Uncensored IQ4_XS (92.39 GiB) + MTP4 + F16 vision — TP=2 tiered experts with a UVA pinned host copy, PLE on ssd-mmap, CED on, 196,608 ctx; state dir + frozen config + payload intact on disk
  • frank-archon-llamacpp-qwen38-flash-next-q8 — RETIRED 2026-09-27 (rollback target): unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X, root unit) serving unsloth Qwen3.8-Flash-Next at Q8_0 (175.3 GiB) on Dankserver's both R9700s — the shared Q8_0 MTP head chained behind ngram-mod, BF16 vision, the experts and the 51B n-gram table computed in host RAM, the placement law, the cutover, and the rollback
  • frank-archon-lucebox-ds4 — RETIRED 2026-09-26 (same-day swap; rollback target): the Lucebox runtime serving DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 40-layer MoE, 384 routed experts top-6, 189 GiB of embedded Engram) on Dankserver's R9700 with the routed experts streamed from the mmapped page cache — held :7770 for hours
  • dankserver-frontier-moe-feasibility — VERIFIED 2026-09-28: can DeepSeek-V4.1-Flash (552B), MiMo-V2.6-Flash-MOPD (309B) or GLM-5.3-Flash (320B) run fast on the 2x R9700 box, and can Affinity / vllm-radiance / FreeToken-AMD be modified for them? The capacity arithmetic (475 GiB payload, 189 GiB of it Engram, ~2.4 GB of cold expert weights per token across PCIe = a ~23 tok/s ceiling) plus the Affinity V4-Flash-to-V4.1 port assessment — reusable expert tiering, four architecture gaps, one a silent-wrongness trap on the Engram key
  • dankserver-qwen38-fp8-expert-offload-tuning — VERIFIED 2026-09-29: the flag-level decode-tuning campaign on the ad hoc qwen-fp8 container (tcclaviger RDNA4 fork, orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8, TP=2 across both R9700s). >35 tok/s MET: 37.44-40.57 tok/s with --expert-prefetch DELETED — telemetry measured the prefetcher at 36% precision, wasting ~half the gather traffic (the LRU itself runs at 82.45% hit); the prefetch-on incumbent 26.8 survived every other challenger (geometry, spec depth, TunableOp-corrupts, batch knobs, concurrency, ngram-fork-bug); the residual gap to ~60 tok/s is the fork's gather read path (6.5-9.5 GB/s effective vs 22.7 at the same slab size)
  • frank-archon-glm53-flash-w4a16 — VERIFIED 2026-09-30: the ACTIVE frank-archon runbook. The cutover to Intel/GLM-5.3-Flash-W4A16-AutoRound (symmetric int4 g128) on the tcclaviger fork, and the reason the int4 AutoRound payload is the ONLY one that serves here (its auto-round quant method routes to the W4A16 offload kernel; a compressed-tensors W4A16 checkpoint of the same model reaches TritonWNA16Experts, which has no offload recipe). Measured decode 9.9-10.0 tok/s / prefill 918 tok/s; the 35 tok/s target is NOT met — the step is bounded by the fork's gather read path (~8.2 GB/s vs a 22.74 GB/s link) with the cache at a 0.53-0.66 hit rate, and MTP4 was measured and rejected (zero decode gain, prefill 918 -> 255). Carries the fork's KV-cap bug, the fp8-KV/gfx12 trap, and the glm47 tool/reasoning parser wiring
  • frank-medium-radiance-mxfp4 — RETIRED 2026-09-26: vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4 W4A8, DFlash2 drafter, unquantized bf16 KV since 2026-09-21) on Dankserver's 2x Radeon AI PRO R9700 behind the :7770 endpoint — the quantization-aware entrypoint, the three-arm battery that chose TP=2, verified capabilities, and the record of the 2026-09-16 two-replica era
  • frank-medium-model-swap — how to swap the checkpoint behind frank-medium: the streamlining toolchain (hf-manifest.py, hf-fetch.sh, swap-frank-medium.sh), the quantization-aware entrypoint contract, the quadlet edit, the deploy, the capability battery, and the rollback path
  • frank-medium-llamacpp-dflash2 — RETIRED 2026-09-16 → 2026-09-20: mainline llama.cpp (gfx1201 HIP build) serving orcarouter Qwen3.8-27B at Q8_0 with the DFlash2 drafter as ONE layer-split process (32:33, two 163,840-token slots); holds the batch/ubatch optimum curve and the per-layer split arithmetic the rollback recipe points back to
  • Dual-R9700-Cooperation — whether the two R9700s can run ONE model: the RCCL/PCIe-atomics evidence on the old TUF GAMING B650-E (which closed tensor parallelism too), what the 2026-09-20 ProArt B850-Creator x8/x8 swap changed — both chains Routing+, both endpoints ReqEn+ — and the measured three-way verdict: radiance TP=2 won and is the deployed shape
  • frank-medium-rocmfpx — PRIOR frank-medium deployment (llama.cpp era): charlie12345/ROCmFPX serving kingjones777/ Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the runtime it replaced
  • freetoken-benchmark-sweep — KV pool vs decode speed sweep (bs1/bs3/bs4/bs6, 8k–700k KV) on the retired Dankserver RTX 3090
  • kingjones-strix-mtp — the Qwen3.8-27B-Q6_0 ROCmFPX AGENT era with the standalone MTP draft (Dankclaw, 2026-08-25 → 2026-08-29; historical)
  • agentionai-fast-imatrix — the retired frank-dragoon runbook: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB, Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30; rollback record)
  • frank-dragoon-halogen27b-dflash2 — RETIRED 2026-10-05 → 10-06 frank-dragoon runbook (root systemd unit + podman, port 7770): the digest-pinned ghcr.io/peonist-ai/halogen image serving peonist-ai/halogen-qwen3.8-27b (qwen3.8-27b-p1w4d-d2.hgn, 33.4 GiB — a .hgn that loads in no other engine) with the DFlash2 drafter on ONE slot of 200,700 ctx (slots>1 disables speculation — engine law); TEXT-ONLY and censored base, both accepted tradeoffs; no server-side sampler, so the thinking recipe rides the wire
  • frank-dragoon-flash-server-v2-native — ACTIVE frank-dragoon runbook since 2026-10-17: halogen-flash-server 0.16.2 serving the NATIVE v2 abliterated conversion of orcarouter's pinned BF16 (qwen38-flash-next-v2-abliterated.hgn, 62.1 GiB) at ctx 200,704 over a 401,408-position pool (HALOGEN_KV_POOL_POSITIONS=401408, 11.0 GiB reserved — one pool behind both slots, raised to 2 × ctx and deployed 2026-10-18 so TWO maximum-length requests coexist) and the stock peonist n-gram/vision companions hardlinked in — engine-tool certified (NIAH 18/18, PPL 3.968, mean KL 0.155); MTP always-on, vision ON, no server-side sampler change
  • frank-dragoon-flash-server-v2-swift15 — RETIRED 2026-10-06 → 10-17 frank-dragoon runbook (rollback target — payload + fetch script on disk): halogen-flash-server 0.16.2 serving the abliterated Swift1.5 finetune of the Qwen3.8-Flash-Next v2 .hgn, two slots over a 524,288-position pool; owns the 400-zone law and the sampler/prompt-cache contract
  • frank-dragoon-rocmfpx-q6agent-uncensored-mtp — RETIRED 2026-10-03 → 10-05 (rollback target; payload, fork build and build script on disk): the pinned kingjones30/ROCmFPX fork (scripts/build-rocmfpx.sh) serving kingjones777/ Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT — ftype 114, 339 tensors of GGML_TYPE_Q6_0_ROCMFPX + 159 type-103, a format no other loader on the host reads — with the repo's Q4_0 MTP head drafting 4 tokens (27.6-28.1 t/s measured, 86.8 % acceptance; the ngram-mod chain measured and rejected) and its BF16 projector, one session at the native 262,144 window
  • frank-dragoon-rocmfpx-q8agent-mtp — RETIRED 2026-10-02 → 10-03 (rollback target, aligned Q8 payload still on disk): the same fork serving the ALIGNED Q8_0_ROCMFPX_AGENT at 17.0 t/s — the +123 % MTP A/B, the traps
  • frank-dragoon-gufo-qwen38-27b — RETIRED 2026-10-01 → 10-02 (history; the payload dirs were removed from disk 2026-10-03, scripts/build-gufo.sh remains): gufo v0.1.1 built from source serving Qwen3.8-27B UD-Q4_K_XL + the z-lab DFlash2 drafter on the gfx1151 iGPU — the two from-source build workarounds, the template-hash allowlist, the measured prefill/decode trade
  • frank-dragoon-vllm-mimo-v26-bf16 — RETIRED 2026-10-01 (same day; replaced by the gufo rollback, then by frank-dragoon-rocmfpx-q8agent-mtp): the digest-pinned kyuz0/vllm-therock-gfx1151 image (vLLM 0.30, TheRock ROCm 10) serving XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16 on the gfx1151 iGPU, 4 x 262,144 ctx, with CROSS-MODEL MTP spec decode from Qwen/Qwen3.5-9B — the roof math (13.1 t/s base is ~94% of LPDDR5x BF16), the measured MTP-vs-DFlash table (19.8 vs 17.4 t/s), the greedy-text numerics trap, the reasoning field change, troubleshooting
  • frank-dragoon-unsloth-qwen38-27b-q8xl — RETIRED 2026-09-28 (replaced by frank-dragoon-gufo-qwen38-27b; rollback target): stock llama.cpp (unsloth b11030, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter (depth 4) on the gfx1151 iGPU — one 262,144-token conversation, q8_0 KV (the build lacks fp8), text-only, Qwen-Sharp v22.4.0 template, the 114 GiB GTT memory model, client contract
  • frank-dragoon-kairic-edge-qwen38-27b — RETIRED 2026-09-21 (replaced by frank-dragoon-unsloth-qwen38-27b-q8xl; same-day swap): the Kairic Edge runtime (ciru-ai/ROCmFPX kairic-edge-qwen38-27b-v1.2, TheRock 7.15.0, Composable Kernel) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (a 27B dense model, main GGUF + FFN/GDN/GDN-Output .pfs sidecars) on the gfx1151 iGPU — one 262,144-token conversation, native MTP depth 4 + n-gram drafting, compatibility mode, the 114 GiB GTT memory model, client contract
  • frank-dragoon-unsloth-qwen38-flash-next — RETIRED 2026-09-21 (replaced by frank-dragoon-kairic-edge-qwen38-27b; same-day swap): the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (a 125B MoE / 6B active) on the gfx1151 iGPU — one 200,000-token conversation, MTP depth 2 via the explicit -md shared-Q8_0 head, the 114 GiB GTT memory model, client contract
  • frank-dragoon-ciru-orca — RETIRED 2026-09-21 (replaced by frank-dragoon-unsloth-qwen38-flash-next; rollback target, 2026-09-19 → 09-21): the CIRU v4.4.1 runtime (private host build, QSA hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU — one 262,144-token conversation, MTP depth 4, 4096 MiB PLE cache, runtime-root gate, client contract
  • halogen-flash-server — RETIRED 2026-09-19 (replaced by frank-dragoon-ciru-orca; rollback target, 2026-09-16 → 09-19): peonist-ai/halogen-flash-server 0.5.8 serving peonist-ai/halogen-qwen3.8- flash-next (w4b .hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one 400,384-position pool, deployed flags, memory model, client contract
  • frank-dragoon-ik-uncensored — the RETIRED 2026-09-16 frank-dragoon runbook and rollback target (2026-09-15 → 09-16): ik_llama.cpp serving mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090 + Strix Halo hybrid — its measured envelope is why the fleet went back
  • dankclaw-iommu-translation-off — Dankclaw's IOMMU boot line: iommu=pt since 2026-10-18 (measured cost of identity DMA: nothing — +2.0 % prefill), the amd_iommu=off era it replaced, and the tracked file + installer that own it
  • kingjones-strix-rocmfp4 — the PRIOR frank-dragoon runbook (2026-08-30 → 2026-09-11) and the rollback target: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF on the card's pinned kingjones30/ROCmFPX build
  • kingjones-q8agent-rocmfpx — the retired Q8_0-AGENT investigation and set_rows failure
  • eemin-rocmi4-mtp — the former caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on Dankclaw
  • ciru-activefpx-promptforge — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
  • kingjones-strix-mtp — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
  • dealign-ornith-crack — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
  • frank-zealot-llama-prism-bonsai9-mtp — the ACTIVE frank-zealot runbook since 2026-10-18: the PrismML vendor llama.cpp build 10777 (the only loader for the custom PQ2_0 tensor type / prism.hadamard.*, and the floor for MTP beside a quantized V cache) serving prism-ml/Ternary-Bonsai-2-9B-gguf PQ2_0 with the MTP head on the RTX 3090 — TWO 262,144-token conversations (--ctx-size 524288 --parallel 2), exact f16 K + f16 V = 16,384 MiB over 8 full-attention layers (32,768 B/token; the per-slot window is UNCHANGED at 262,144 — what halved is the POOL, because exact f16 over the OLD 786,432 pool bills 24,576 MiB against the 24,123 the driver leaves addressable; the old "26,112 vs 13,056 MiB" note doubled the q8_0 figure with its 1/32 scale overhead included, so f16/q8_0 is 1.882×, not 2×), MTP off by measurement (the draft context bills 2,048 MiB of f16 KV over the whole pool, and exact KV leaves no room for one), vision through the Q8_0 mmproj; boot 17,876 used / 6,249 free (the q8_0 era)
  • frank-zealot-llama-prism-bonsai2-pq20 — RETIRED 2026-10-17 → 10-18 (rollback target, payload + build 10754 on disk): Ternary-Bonsai-2-27B PQ2_0 at TWO 200,704-token conversations, q8_0 K + q8_0 V, MTP impossible (no head in that export), vision through the Q8_0 mmproj; boot 22,664 / 1,461
  • frank-zealot-llama-qwen35-9b-q6k-2x200704 — RETIRED 2026-10-17 (rollback target, payload on disk): llama.cpp master (CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive Q6_K on the RTX 3090 — TWO 200,704-token conversations, exact f16 K + f16 V, MTP impossible (the file carries no nextn/mtp tensors), vision through the publisher's BF16 mmproj; priced from the MiMo Q6_K boots (same qwen35 geometry)
  • frank-zealot-llama-gemma4-q4km-200704 — RETIRED 2026-10-07 (same day; payload on disk as rollback): llama.cpp master (CUDA sm_86) serving HauhauCS Gemma4-26B-A4B QAT Uncensored Q4_K_M on the RTX 3090 — ONE 200,704-token conversation, f16 K + q8_0 V, MTP disabled (the gemma4-assistant drafter costs ~1.9 GiB: boot was 23,806 used/319 free with it, 21,890/2,235 without), vision through the publisher's mmproj
  • frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688 — RETIRED 2026-10-07 (superseded by gemma4; payload on disk as the rollback): the frank-zealot runbook 2026-10-05 (evening) → 2026-10-07 (UNCECERSED RCO-IQ3_XXS with MTP back on since 2026-10-06 evening; renamed from frank-small and reshaped to TWO 114,688-token conversations at Frank's 112k cap with the ubatch ladder climbed to 2048 late 2026-10-06): llama.cpp master (CUDA sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS (the MTP head in the file is drafted) on the RTX 3090 — 1,005 MiB free at boot (763 settled after a vision probe), q8_0 K + q8_0 V, vision through the publisher's own projector, ~52–58 t/s recipe decode with MTP
  • frank-zealot-llama-mimo-q6k-2x200704 — RETIRED 2026-10-05 (evening, superseded by swift15): llama.cpp master serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q6_K — TWO conversations at 200,704 tokens each with fully unquantized f16 KV and mmproj vision; 94.4 tok/s decode (the Q6_K rung costs a measured −10 % vs Q5_K_M — weight-bandwidth), batch 4096 / ubatch 2048 (+4.6 % prefill, measured 2026-10-05), 2,279 MiB free, and the ≥256 px image-probe trap (tiny images answer "video")
  • frank-zealot-llama-mimo-q5km — RETIRED 2026-10-02 → 10-03 (rollback target, payload on disk; owns the measured boot-pair table): same runtime and shape one rung lower — 104.8 tok/s, 3,571 MiB true free
  • frank-zealot-llama-mimo-q6k — RETIRED 2026-10-01 (same day; that same Q6_K rung returned ACTIVE 2026-10-03 at the new shape, see frank-zealot-llama-mimo-q6k-2x200704): the first llama.cpp + GGUF shape for this endpoint, Q6_K at 2 x 262,144 / f16-K + q8_0-V, 20.9 GiB measured
  • frank-zealot-sglang-mimo-v26 — RETIRED 2026-10-01 (rollback target): SGLang in a self-contained python3.12 venv serving letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, INT8 W8A16, in-checkpoint vision, NO MTP weights shipped) on Dankclaw's RTX 3090, port 7771 — fp8 KV, pinned 606,208-token pool, 2 x 262,144 slots, bf16 mamba states, eager prefill
  • frank-zealot-gemma4-12b — RETIRED 2026-10-01 (rollback target): mainline llama.cpp serving HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M (MTP assistant sidecar + ngram-mod, q8_0 KV) on Dankclaw's RTX 3090, port 7771
  • frank-zealot-vllm-gemma4 — RETIRED 2026-09-19 (same-day revert, rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16 Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647- token pool, 2 concurrent full-context sessions, native text+image+audio
  • frank-zealot-qwen38-gsq-rco — RETIRED 2026-09-17 (rollback target): the prior frank-zealot runbook (2026-09-16 → 09-17), ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with own in-file MTP head + BF16 projector
  • ikllama-ornith-dankserver — the retired ik_llama.cpp runtime for the old frank-zealot (q8_0 + Hadamard KV cache, split-mode none; 2026-08-26)
  • freetoken-frank-small — RETIRED 2026-09-15: the FreeToken/CUDA frank-small runbook, retired with the RTX 3090
  • kairic-edge — the retired Kairic Edge IU4 experiment (2026-08-22)
  • dflash2-large-vulkan — the retired DFlash2 Vulkan deployment (2026-08-21)
  • Dankserver System Specs — the 2x R9700 box hardware (frank-medium)
  • Dankclaw System Specs — the Strix Halo gfx1151 box (frank-dragoon)
  • Dankclaw Improvements — host-level changes on the Strix Halo box
  • Swap Configuration — the 64 GiB /swap.img host swapfile (raised from 8 GiB 2026-09-12): fstab-driven swap.img.swap unit, the safe resize ordering that drains the old file into fresh swap instead of into a host whose MemAvailable lies by the size of the model, and the swappiness caveat
  • frank-npu — the RETIRED FLM NPU endpoint on Dankclaw (2026-09-11): Gemma4 E4B vision+audio pack on the XDNA2 NPU, port 7771; kept as the retirement record, including the IOMMU/SVA err=-19 root cause and where every dependent route moved

Benchmarks and Sizing

  • frank-dragoon-benchmark — backend/spec benchmark behind the old head
  • ubatch-benchmark-large — active-model llama-bench ubatch sweep plan and results
  • frank-zealot-benchmark — the small-model benchmark suite
  • frank-medium-batch-ubatch — the 2026-09-16 batch/ubatch sweep on the 2x R9700 layer split: prefill PEAKS at ubatch 512 and falls on both sides, the logical batch buys nothing measurable, decode never moves, and the ubatch's uneven VRAM give-back (1.30 GiB on card 0 against 2.41 GiB on card 1) moved the tensor split from 36:29 to 35:30
  • Running Qwen3.6 35B A3B FP8 on Dankserver — Qwen3.6-35B-A3B hardware notes
  • Running Qwen3.6 27B FP8 on Dankclaw — Qwen3.6-27B sizing on the iGPU
  • qwen3.8-strix-optimizations — Strix Halo tuning notes
  • x4-pair-experiment — what a 3090 + Strix Halo combination can and cannot do in mainline llama.cpp (measured 2026-09-16): the tensor inventory that decides a split, the one flag (-ot) that kills per-tensor placement across devices, the two-device layer splits that DO work (pure-Vulkan and cross-vendor CUDA+Vulkan, +7 % decode at 64k), and the harness-parsing trap that made the first round of measurements say otherwise

Memory and Hardware

Retired Topology and History

vLLM Reference (historical)

Pipeline Parallelism (archive)

Update Log

  • log — chronological record of knowledge-base changes

See Also

Reference in New Issue View Git Blame Copy Permalink