dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

llm: retire the trio service names — fleet services renamed to frank-archon/frank-dragoon/frank-zealot

Browse Source 
The 2026-10-09 wire aliases become the service names everywhere:
systemd units, serve scripts, env files and FRANK_* variables,
deploy-plan/deploy-remote service keys, container names, Prometheus
job names + alert rules, Grafana dashboard queries, bats seams,
runbook filenames, homepage/AGENTS lore, and historical prose.
Rollback tags renamed in lockstep. Residue in the tree: zero.
Retired frank-medium/small/npu names are history and stay.

Known casualty: scripts/patches/*.patch context lines were rewritten;
those retired-era vLLM-fork patches no longer apply byte-exact.
This commit is contained in:
Frank
2026-10-10 17:03:14 -07:00
parent 5ced474eb9
commit 55f87fcdce
195 changed files with 2085 additions and 2064 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/index.md
+67 -67
View File
@@ -1,6 +1,6 @@
# LLM Fleet — Knowledge Base
**Dankserver runs the radiance engine (StillDeadcode) on frank-primary**
**Dankserver runs the radiance engine (StillDeadcode) on frank-archon**
(digest-pinned `stilldeadcode/radiance` image, engine as entrypoint,
driving `podman run` directly, root systemd unit, :7770): the OFFICIAL
`Qwen/Qwen3.8-Flash-Next-FP8` checkpoint at NATIVE block-FP8 precision
@@ -8,15 +8,15 @@ driving `podman run` directly, root systemd unit, :7770): the OFFICIAL
the descriptor-staging law), on BOTH R9700s TP=2 (`--tp-wire wht6`),
experts in a 40 GiB/rank pinned host pool + VRAM cache (ratio 0.82), the
47.7 GiB PLE table pinned to host RAM, fp8 KV, MTP spec decode depth 3,
2 × 200,000 first boot. Rollback tags frank-primary-freetoken-pr587-20261005
and frank-primary-vllm-shapeG-20261003),
2 × 200,000 first boot. Rollback tags frank-archon-freetoken-pr587-20261005
and frank-archon-vllm-shapeG-20261003),
and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2
(frank-large, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned
(frank-dragoon, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned
digest serving the Quat3rnion ABLITERATED Swift1.5 finetune of
Qwen3.8-Flash-Next v2, a ~62 GiB v2 .hgn that loads in no other engine,
MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots at 200,192
ctx, vision ON)** and
**llama.cpp CUDA (frank-secondary, port 7771: HauhauCS Gemma4-26B-A4B QAT
**llama.cpp CUDA (frank-zealot, port 7771: HauhauCS Gemma4-26B-A4B QAT
Uncensored Q4_K_M GGUF — 26B MoE hybrid (25 sliding-window + 5 full-attention
layers), ONE 200,704-token conversation, MTP disabled,
q8_0 K + q8_0 V (FP8 class), vision via the publisher's own BF16
@@ -35,7 +35,7 @@ graph LR
A[2x Radeon AI PRO R9700
gfx1201, 32 GiB each]
B2[radiance StillDeadcode, digest-pinned image
podman run, frank-primary, port 7770]
podman run, frank-archon, port 7770]
C[Qwen3.8-Flash-Next FP8 OFFICIAL, native block-FP8
2x R9700 TP=2 wht6, experts 40 GiB/rank pinned pool + VRAM,
PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 3,
@@ -46,11 +46,11 @@ PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 3,
E2[RTX 3090 (PCIe x4 dock) +
gfx1151 iGPU 114 GiB GTT]
E[flash-server container gfx1151, digest-pinned image
frank-large, port 7770]
frank-dragoon, port 7770]
F[flash-server v2 abliterated Swift1.5 .hgn + ngram table + vision tower
2 x 200192 ctx, MTP always-on, image input]
G[llama.cpp master CUDA sm_86
frank-secondary, port 7771]
frank-zealot, port 7771]
H[Gemma4-26B-A4B QAT Uncensored Q4_K_M GGUF
1 x 200704 ctx, q8_0 K + q8_0 V, MTP off, mmproj vision]
E2 --> E --> F
@@ -71,7 +71,7 @@ frank-secondary, port 7771]
## Active Deployment
- [[frank-large-flash-server-v2-swift15]] — **ACTIVE since 2026-10-06**
- [[frank-dragoon-flash-server-v2-swift15]] — **ACTIVE since 2026-10-06**
(root systemd unit + podman, port 7770): the digest-pinned
`ghcr.io/peonist-ai/halogen-flash-server` 0.16.2 engine serving the
Quat3rnion ABLITERATED Swift1.5 finetune of peonist's
@@ -88,7 +88,7 @@ frank-secondary, port 7771]
greedy, ~1,640 t/s prefill at 34k, ~72 % fewer thinking tokens.
Rollback: revert the swap commit; the 27B payload + fetch script stay
on disk
- [[frank-large-halogen27b-dflash2]] — **RETIRED 2026-10-05 → 10-06
- [[frank-dragoon-halogen27b-dflash2]] — **RETIRED 2026-10-05 → 10-06
(rollback target; payload + fetch script + image on disk)**: the
digest-pinned `ghcr.io/peonist-ai/halogen` 0.1.4 engine serving
`peonist-ai/halogen-qwen3.8-27b` (`qwen3.8-27b-p1w4d-d2.hgn`, 33.4 GiB)
@@ -99,24 +99,24 @@ frank-secondary, port 7771]
the FA tile rejection (`HALOGEN_FA_TQ=256`: prefill −9.5 %, decode
±0.3 %), and deep decode at 3.3-3.8 t/s@80k being structural. The
foreground-`podman run` ledger capture and the deep-role rerouting
(`slow`/`advisor` → frank-primary) carried forward. Superseded by the
(`slow`/`advisor` → frank-archon) carried forward. Superseded by the
MODEL swap: the abliterated v2 .hgn loads only in flash-server, so the
engine moved with the model
- [[frank-secondary-llama-qwen35-9b-q6k-2x200704]] — **ACTIVE since 2026-10-07**
- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — **ACTIVE since 2026-10-07**
(root systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** on the RTX 3090
— TWO 200,704-token conversations (`--ctx-size 401408 --parallel 2`), exact
f16 K + f16 V, vision via the publisher's BF16 mmproj. MTP is OFF because
the file has no MTP tensors at all. Priced from the MiMo Q6_K boots (same
qwen35 geometry, same card, same shape). Supersedes the Gemma4 era (below).
- [[frank-secondary-llama-gemma4-q4km-200704]] — **RETIRED 2026-10-07** (same
- [[frank-zealot-llama-gemma4-q4km-200704]] — **RETIRED 2026-10-07** (same
day; superseded by qwen35-9b-q6k; payload on disk as rollback): llama.cpp
master (CUDA sm_86) serving HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-
HauhauCS-Balanced-MTP **Q4_K_M** on the RTX 3090 — ONE 200,704-token
conversation, f16 K + q8_0 V, vision via the publisher's mmproj. MTP was
OFF (the gemma4-assistant drafter costs ~1.9 GiB: 23,806 used/319 free with
it, 21,890/2,235 without).
- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — **RETIRED
- [[frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688]] — **RETIRED
2026-10-07** (superseded by gemma4-q4km; payload on disk as rollback);
ACTIVE 2026-10-05 (evening) → 2026-10-07; UNCECERSED RCO-IQ3_XXS with
MTP back on since
@@ -139,7 +139,7 @@ frank-secondary, port 7771]
98/100). Replaces BOTH same-day payloads: the live MiMo Q6_K shape
(below) and the Ornith-1.5-9B Q8_0 swap that was committed
(77decb5..380aafa) but never deployed
- [[frank-secondary-llama-mimo-q6k-2x200704]] — **RETIRED 2026-10-05 (evening;
- [[frank-zealot-llama-mimo-q6k-2x200704]] — **RETIRED 2026-10-05 (evening;
rollback target; payload on disk)**: llama.cpp master (CUDA sm_86)
serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, TWO slots of
200,704 with FULLY UNQUANTIZED KV (f16 K + f16 V) and the mmproj for
@@ -148,15 +148,15 @@ frank-secondary, port 7771]
(+4.6 % prefill, measured), 2,279 MiB of the card left free. This was
the LIVE payload when Swift cut over. The ubatch A/B table and the
probe-size trap (>=256 px images or the model says "video") live inside
- [[frank-secondary-llama-mimo-q5km]] — **RETIRED 2026-10-02 → 10-03 (rollback
- [[frank-zealot-llama-mimo-q5km]] — **RETIRED 2026-10-02 → 10-03 (rollback
target; payload on disk; owns the measured boot-pair table)**: the same
runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower —
104.8 tok/s, 3,571 MiB true free
- [[frank-primary-radiance-swift15-w4dn8]] — **STAGED (campaign running)**:
- [[frank-archon-radiance-swift15-w4dn8]] — **STAGED (campaign running)**:
Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` on radiance
1.0.13 (recipe + 10M-token calibration + KLD gate vs the BF16 container);
replaces the FP8-native payload when the gates pass
- [[frank-primary-radiance-qwen38-native-fp8]] — **ACTIVE since 2026-10-05**
- [[frank-archon-radiance-qwen38-native-fp8]] — **ACTIVE since 2026-10-05**
(root systemd unit, port 7770): the StillDeadcode **radiance** engine
(C++/HIP, libr4d gfx1201 kernels, digest-pinned 287 MiB image, engine as
the image entrypoint) serving the OFFICIAL `Qwen/Qwen3.8-Flash-Next-FP8`
@@ -171,12 +171,12 @@ frank-secondary, port 7771]
ram`), fp8 KV 7.32 KiB/token, MTP spec decode depth 3, 2 × 200,000 first
boot, froggeric v22.5 template, vision + tools + `reasoning_content` auto,
`/metrics` scraped. Rollback: git tag
`frank-primary-freetoken-pr587-20261005` (FreeToken image + orcarouter
payload on disk); older rollback `frank-primary-vllm-shapeG-20261003`
- [[frank-primary-freetoken-pr587]] — **RETIRED 2026-10-05 (same day;
`frank-archon-freetoken-pr587-20261005` (FreeToken image + orcarouter
payload on disk); older rollback `frank-archon-vllm-shapeG-20261003`
- [[frank-archon-freetoken-pr587]] — **RETIRED 2026-10-05 (same day;
rollback target)**: FreeToken at PR #587 head `77a7201` (branch
rocm714_torch214) built on the host into the local image
`frank-primary-freetoken:pr587-77a7201` (Dockerfile: digest-pinned
`frank-archon-freetoken:pr587-77a7201` (Dockerfile: digest-pinned
rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev-symlink law,
curated --no-deps install per docs/install_amd.md), serving the SAME
orcarouter FP8 payload on ONE R9700 — experts in pinned host RAM
@@ -198,8 +198,8 @@ frank-secondary, port 7771]
and a mounted compile cache. Why `--max-num-seqs`
must stay 3 (SEQS 2 cannot compile this model), why the arena's levers are
`max_model_len` and the overhead term rather than the KV pool, and the measured envelope
- [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro):
- [[frank-archon-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
failed GLM experiment; frank-archon runs Qwen FP8 again, see the intro):
the tcclaviger RDNA4 vLLM fork
(digest-pinned `tcclaviger/vllm@sha256:da4029666…`, plain `podman run`,
no shim) serving Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric
@@ -215,10 +215,10 @@ frank-secondary, port 7771]
35 tok/s target is NOT met, bounded by the fork's expert-gather read path
(~8.2 GB/s effective vs a 22.74 GB/s link) at a 0.53-0.66 cache hit rate.**
Rollback: commit `2339fae` (the qwen FP8 + MTP4 shape of 2026-09-29 → 09-30)
- [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
- [[frank-archon-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
back; payload and R9V state dirs intact on disk, git tag
frank-primary-r9v-q8-20260929 is the last pre-cutover commit):
[[frank-primary-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
frank-archon-r9v-q8-20260929 is the last pre-cutover commit):
[[frank-archon-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
(same-day swap to the Q8_0 payload; rollback target, state + payload
intact on disk): the R9V radiance-vLLM fork serving
Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) +
@@ -226,8 +226,8 @@ frank-secondary, port 7771]
experts (UVA pinned host copy, 62 hot + 160-slot LRU on rank 0, 428
static on rank 1), PLE n-gram on ssd-mmap, CED on, 196608 ctx. Cut over
from llama.cpp the same day; rollback target
[[frank-primary-llamacpp-qwen38-flash-next-q8]]
- [[frank-primary-llamacpp-qwen38-flash-next-q8]] — RETIRED 2026-09-27
[[frank-archon-llamacpp-qwen38-flash-next-q8]]
- [[frank-archon-llamacpp-qwen38-flash-next-q8]] — RETIRED 2026-09-27
(rollback target; the llama.cpp b11030 endpoint of 2026-09-26):
unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X, bundled runtime)
serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 across both R9700s —
@@ -236,7 +236,7 @@ frank-secondary, port 7771]
(depth 2) chained behind ngram-mod, BF16 vision, one full 262,144-token
conversation, `reasoning_content`, the placement law, the cutover and
rollback
- [[frank-primary-lucebox-ds4]] — RETIRED 2026-09-26 (same-day swap,
- [[frank-archon-lucebox-ds4]] — RETIRED 2026-09-26 (same-day swap,
rollback target; active for hours): the Lucebox DS4.1 engine
(localhost/lucebox-rocm:gfx1201) serving
Lucebox/DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB) with streamed experts —
@@ -259,8 +259,8 @@ frank-secondary, port 7771]
ngram-fork-bug); the residual gap to ~60 tok/s is the fork's gather
read path (6.5-9.5 GB/s effective vs 22.7 at the same slab size)
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26 (replaced by
[[frank-primary-lucebox-ds4]], then
[[frank-primary-llamacpp-qwen38-flash-next-q8]]): vllm-radiance 0.9.3
[[frank-archon-lucebox-ds4]], then
[[frank-archon-llamacpp-qwen38-flash-next-q8]]): vllm-radiance 0.9.3
serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark
W4A8) with the z-lab DFlash2-FP8 drafter as ONE tensor-parallel process
(TP=2) behind :7770, on unquantized bf16 KV (`KV_DTYPE=auto`, cut
@@ -288,15 +288,15 @@ frank-secondary, port 7771]
kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the
runtime it replaced
- [[freetoken-frank-small]] — RETIRED 2026-09-15, when the RTX 3090 moved to
frank-large: the FreeToken v0.1.2 frank-secondary runbook (batsclamp
frank-dragoon: the FreeToken v0.1.2 frank-zealot runbook (batsclamp
Huihui-Qwen3.6-35B-A3B-abliterated-FP8), kept as the retirement record. The
NAME was reused on 2026-09-16 by a different service on Dankclaw — see
[[frank-secondary-qwen38-gsq-rco]]
[[frank-zealot-qwen38-gsq-rco]]
- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
- [[deployment]] — deploy and update process for both servers
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[frank-large-rocmfpx-q6agent-uncensored-mtp]] — **RETIRED 2026-10-03 →
- [[frank-dragoon-rocmfpx-q6agent-uncensored-mtp]] — **RETIRED 2026-10-03 →
10-05 (rollback target; payload, fork build and build script on disk)**:
the pinned ROCmFPX fork serving the uncensored protected-heads
Q6_0_ROCMFPX_AGENT rung with the repo's MTP-Q4_0 head drafting 4 tokens,
@@ -304,21 +304,21 @@ frank-secondary, port 7771]
acceptance. Replaced by the halogen DFlash2 endpoint; the ngram-mod-onto-MTP
rejection (-15 %, recurrent-state rollback) stays a live lesson for this
architecture.
- [[frank-large-rocmfpx-q8agent-mtp]] — **RETIRED 2026-10-02 → 10-03 (the
- [[frank-dragoon-rocmfpx-q8agent-mtp]] — **RETIRED 2026-10-02 → 10-03 (the
rollback target; payload still on disk)**: the same pinned ROCmFPX fork
serving the ALIGNED `Q8_0_ROCMFPX_AGENT` (ftype 115, 17.0 t/s, MTP
acceptance 0.462) — the MTP-vs-none +123 % A/B, the trap catalogue
- [[frank-large-gufo-qwen38-27b]] — **RETIRED 2026-10-02 (history; the
- [[frank-dragoon-gufo-qwen38-27b]] — **RETIRED 2026-10-02 (history; the
payload dirs were removed from disk 2026-10-03, so this rollback is a
re-download)** (root systemd unit, port 7770): gufo
(gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from
source on Dankclaw, serving unsloth/Qwen3.8-27B-GGUF **UD-Q4_K_XL**
with the z-lab DFlash2 Q4_K_M drafter — ONE session at the full native
262,144 window, unquantized f16 KV. Rolled back by Frank after one day
of [[frank-large-vllm-mimo-v26-bf16]]. The two from-source build
of [[frank-dragoon-vllm-mimo-v26-bf16]]. The two from-source build
workarounds, the client-contract delta, the measured prefill/decode
trade against llama.cpp
- [[frank-large-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (one day;
- [[frank-dragoon-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (one day;
rollback by Frank after the autoresearch loop): the digest-pinned
kyuz0/vllm-therock-gfx1151 image serving
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16, 4 x 262144 ctx,
@@ -327,68 +327,68 @@ frank-secondary, port 7771]
NEGATIVE on real chat+reasoning traffic), the roof math, the greedy-text
numerics trap, and the death of every ≥6-bit vLLM quant path on gfx1151
(quark int8 rejected, FP8 kernels CDNA-only, torchao unwired)
- [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
[[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp
- [[frank-dragoon-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
[[frank-dragoon-gufo-qwen38-27b]]; rollback target): stock llama.cpp
(unsloth b11030, ROCm gfx1151) serving
unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2
Q4_K_M drafter (depth 4) on the gfx1151 iGPU — one full 262,144-token
conversation, q8_0 KV (the build lacks fp8), text-only, Qwen-Sharp
v22.4.0 template, the 114 GiB GTT memory model, client contract,
troubleshooting
- [[frank-large-kairic-edge-qwen38-27b]] — RETIRED 2026-09-21 (same-day
swap, replaced by [[frank-large-unsloth-qwen38-27b-q8xl]]; rollback
- [[frank-dragoon-kairic-edge-qwen38-27b]] — RETIRED 2026-09-21 (same-day
swap, replaced by [[frank-dragoon-unsloth-qwen38-27b-q8xl]]; rollback
target): the Kairic Edge runtime (ciru-ai/ROCmFPX v1.2, built from
source) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (IU4 W4A4, 27B dense)
on the gfx1151 iGPU — one full 262,144-token conversation, MTP depth 4 +
n-gram drafting, the three PromptForge sidecars, compatibility mode, the
114 GiB GTT memory model, client contract, troubleshooting
- [[frank-large-unsloth-qwen38-flash-next]] — RETIRED 2026-09-21 (same-day
swap, replaced by [[frank-large-kairic-edge-qwen38-27b]]; rollback target):
- [[frank-dragoon-unsloth-qwen38-flash-next]] — RETIRED 2026-09-21 (same-day
swap, replaced by [[frank-dragoon-kairic-edge-qwen38-27b]]; rollback target):
the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving
unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (125B MoE) on the gfx1151 iGPU —
one 200,000-token conversation, MTP depth 2 via the explicit -md shared-Q8_0
head, the 114 GiB GTT memory model, client contract, troubleshooting
- [[frank-large-ciru-orca]] — RETIRED 2026-09-21 (replaced by
[[frank-large-unsloth-qwen38-flash-next]]; rollback target): the CIRU
- [[frank-dragoon-ciru-orca]] — RETIRED 2026-09-21 (replaced by
[[frank-dragoon-unsloth-qwen38-flash-next]]; rollback target): the CIRU
v4.4.1 runtime (private host build, QSA hotfix) serving
jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU — one
262,144-token conversation, MTP depth 4, 4096 MiB PLE cache, runtime-root
gate, client contract, troubleshooting (2026-09-19 → 09-21)
- [[halogen-flash-server]] — RETIRED 2026-09-19 (replaced by
[[frank-large-ciru-orca]]; rollback target): peonist-ai halogen-flash-server
[[frank-dragoon-ciru-orca]]; rollback target): peonist-ai halogen-flash-server
0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b
.hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
400,384-position pool, deployed flags, memory model, client contract,
troubleshooting (2026-09-16 → 09-19)
- [[frank-secondary-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02** (the same
- [[frank-zealot-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02** (the same
rung returned ACTIVE 2026-10-03 at the 2 x 200,704 f16-KV shape —
[[frank-secondary-llama-mimo-q6k-2x200704]]): llama.cpp (master, CUDA sm_86
[[frank-zealot-llama-mimo-q6k-2x200704]]): llama.cpp (master, CUDA sm_86
build) serving
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO
native 262,144-token sessions with ASYMMETRIC KV (K f16 unquantized + V
q8_0, which forces flash-attn), ~20.9/24.6 GiB measured, text-only (the
GGUF has no vision), reasoning_content, no spec decode; payload still at
`/home/dank/models/MiMo-Q6K/`
- [[frank-secondary-sglang-mimo-v26]] — RETIRED 2026-10-01 (hours; replaced
by [[frank-secondary-llama-mimo-q6k]] for the asymmetric KV SGLang cannot
- [[frank-zealot-sglang-mimo-v26]] — RETIRED 2026-10-01 (hours; replaced
by [[frank-zealot-llama-mimo-q6k]] for the asymmetric KV SGLang cannot
express; venv still on disk, INT8 payload staged to
`~dank/.AGENTTRASH/20261002-retired-llm-payloads/`): SGLang in a
self-contained python3.12 venv serving leteschlead MiMo-V2.6 INT8
W8A16 AutoRound, 2 x 262144 ctx, fp8 KV, in-checkpoint vision,
83.9 tok/s decode
- [[frank-secondary-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by
[[frank-secondary-sglang-mimo-v26]] → the llama.cpp MiMo shape; payload staged
- [[frank-zealot-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by
[[frank-zealot-sglang-mimo-v26]] → the llama.cpp MiMo shape; payload staged
to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/`):
mainline llama.cpp serving
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
3090, with the separate MTP assistant sidecar chained behind ngram-mod,
q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB.
- [[frank-secondary-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
- [[frank-zealot-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-
token pool, 2 concurrent full-context sessions, native text+image+audio.
- [[frank-secondary-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the
prior frank-secondary runbook (2026-09-16 → 09-17), ISTA-DASLab
- [[frank-zealot-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the
prior frank-zealot runbook (2026-09-16 → 09-17), ISTA-DASLab
Qwen3.8-27B-GSQ-RCO IQ3_S with the model's own in-file MTP head and
f16 KV at 131k
- [[x4-pair-experiment]] — MEASURED 2026-09-16: what mainline llama.cpp can and
@@ -398,7 +398,7 @@ frank-secondary, port 7771]
that work (+7 % decode at 64k, cross-vendor included), the harness-parsing trap
that invalidated the first round, and the iGPU-only result that matches
halogen's deployed decode rate — at a memory cost the co-tenants cannot pay
- [[frank-large-ik-uncensored]] — RETIRED 2026-09-16 (rollback target): the
- [[frank-dragoon-ik-uncensored]] — RETIRED 2026-09-16 (rollback target): the
ik_llama.cpp runbook (root systemd unit, 2026-09-15 → 09-16) serving
mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090
(dense/attention/KV) + LPDDR (experts/PLE) hybrid; its measured envelope and
@@ -407,7 +407,7 @@ frank-secondary, port 7771]
`amd_iommu=off` since 2026-09-11 (13-16 % prefill, measured upstream),
what it cost (the NPU endpoint), what was deliberately not copied from
the reference machine, and how to revert
- [[kingjones-strix-rocmfp4]] — PRIOR frank-large runbook and rollback
- [[kingjones-strix-rocmfp4]] — PRIOR frank-dragoon runbook and rollback
target: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params
(56B MoE + 51.2B PLE), single 113.47 GiB GGUF, on the card's pinned
kingjones30/ROCmFPX fork (2026-08-30 → 2026-09-11)
@@ -419,9 +419,9 @@ frank-secondary, port 7771]
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model
on Dankclaw (retired)
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on
Dankserver (frank-secondary, retired 2026-09-02)
Dankserver (frank-zealot, retired 2026-09-02)
- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for
frank-secondary (2026-08-26): q8_0 + Hadamard KV cache,
frank-zealot (2026-08-26): q8_0 + Hadamard KV cache,
split-mode none (single-GPU), mandatory FA_ALL_QUANTS build flags
- [[kingjones-strix-mtp]] — the former Q6 AGENT runbook (restored 2026-09-01
after Q8_0 failure, then superseded by the STRIX deployment on 2026-08-27)
@@ -429,9 +429,9 @@ frank-secondary, port 7771]
on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
(deployed and reverted 2026-08-22)
- [[Dankserver System Specs]] — the 2x R9700 box (frank-primary)
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on
the flash-server v2 runtime since 2026-10-06, frank-secondary on the RTX 3090)
- [[Dankserver System Specs]] — the 2x R9700 box (frank-archon)
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-dragoon on
the flash-server v2 runtime since 2026-10-06, frank-zealot on the RTX 3090)
- [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint
on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as
the retirement record — the IOMMU/SVA err=-19 root cause and where
@@ -455,7 +455,7 @@ frank-secondary, port 7771]
- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware
notes and sizing (this architecture family is what the retired frank-secondary
notes and sizing (this architecture family is what the retired frank-zealot
served in the Huihui abliterated FP8 variant — see
[[freetoken-frank-small]]; the Ornith-1.5-35B-A3B CRACK checkpoint
that preceded it is archived at [[dealign-ornith-crack]])
@@ -471,7 +471,7 @@ frank-secondary, port 7771]
- [[Qwen3.8 PP Deployment]] — the retired pipeline-parallel head + worker
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history (superseded by FastMTP)
- [[FastFlowLM NPU Inference]] — the retired FastFlowLM NPU endpoint
- [[frank-large-benchmark]] — backend/spec benchmark behind the old head
- [[frank-dragoon-benchmark]] — backend/spec benchmark behind the old head
- [[dankserver-vram-growth]] — VRAM/RAM growth behind the retired cuBLAS crash
- [[DFlash DSpark Speculative Decoding]] — retired DFlash/DSpark drafting
- [[Quantization by Hardware Architecture]] — which quant format fits which GPU/CPU
@@ -529,7 +529,7 @@ These sit alongside the operational pages under
## Related
- [Project knowledge base](../index.md) — the Podman app stacks on the same
two hosts. OpenWebUI (Dankclaw) consumes the on-host `frank-large`
two hosts. OpenWebUI (Dankclaw) consumes the on-host `frank-dragoon`
endpoint at `:7770`. The Gitea instance on Dankclaw hosts this repository.
- [Deployment Flow](../Architecture/Deployment%20Flow.md) — the service-scoped
git-based deploy with inference isolation