Frank
41d2f56269
23,120 used / 1,005 free at boot, 763 settled after a vision probe (encoder buffers resident). Rung cost 746 MiB, not the transferable 616 prediction. Prefill baseline 1,209 t/s @ 3,760 recorded; decode unchanged as the law says. 2048 = ceiling for this shape.
513 lines
31 KiB
Markdown
513 lines
31 KiB
Markdown
# LLM Fleet — Knowledge Base
|
||
|
||
**Dankserver runs the radiance engine (StillDeadcode) on frank-primary**
|
||
(digest-pinned `stilldeadcode/radiance` image, engine as entrypoint,
|
||
driving `podman run` directly, root systemd unit, :7770): the OFFICIAL
|
||
`Qwen/Qwen3.8-Flash-Next-FP8` checkpoint at NATIVE block-FP8 precision
|
||
(converted no-recipe to `qwen4exp-fp8-native.rad` — see the runbook for
|
||
the descriptor-staging law), on BOTH R9700s TP=2 (`--tp-wire wht6`),
|
||
experts in a 40 GiB/rank pinned host pool + VRAM cache (ratio 0.82), the
|
||
47.7 GiB PLE table pinned to host RAM, fp8 KV, MTP spec decode depth 3,
|
||
2 × 200,000 first boot. Rollback tags frank-primary-freetoken-pr587-20261005
|
||
and frank-primary-vllm-shapeG-20261003),
|
||
and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2
|
||
(frank-large, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned
|
||
digest serving the Quat3rnion ABLITERATED Swift1.5 finetune of
|
||
Qwen3.8-Flash-Next v2, a ~62 GiB v2 .hgn that loads in no other engine,
|
||
MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots at 200,192
|
||
ctx, vision ON)** and
|
||
**llama.cpp CUDA (frank-secondary, port 7771: ajgazin Swift-1.5-Qwen3.8-27B
|
||
Uncensored Dynamic-MTP RCO-IQ3_XXS GGUF — 27B (the MTP head in the file
|
||
IS drafted, draft-mtp depth 3), TWO 131,072 conversations at Frank's
|
||
128k cap (the tightest fit in the endpoint's history — 489 MiB free),
|
||
q8_0 K + q8_0 V (FP8 class), vision via the publisher's own BF16
|
||
mmproj, on the RTX 3090)**.
|
||
|
||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||
> `serverAppConfigs-docs/` root to `serverAppConfigs-docs/LLM/`. The fleet scripts and units now live in the
|
||
> repo's `scripts/` directory.
|
||
|
||
## Deployment Topology
|
||
|
||
```mermaid
|
||
graph LR
|
||
subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
|
||
A[2x Radeon AI PRO R9700
|
||
gfx1201, 32 GiB each]
|
||
B2[radiance StillDeadcode, digest-pinned image
|
||
podman run, frank-primary, port 7770]
|
||
C[Qwen3.8-Flash-Next FP8 OFFICIAL, native block-FP8
|
||
2x R9700 TP=2 wht6, experts 40 GiB/rank pinned pool + VRAM,
|
||
PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 3,
|
||
2 x 200000, froggeric v22.5]
|
||
end
|
||
|
||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||
E2[RTX 3090 (PCIe x4 dock) +
|
||
gfx1151 iGPU 114 GiB GTT]
|
||
E[flash-server container gfx1151, digest-pinned image
|
||
frank-large, port 7770]
|
||
F[flash-server v2 abliterated Swift1.5 .hgn + ngram table + vision tower
|
||
2 x 200192 ctx, MTP always-on, image input]
|
||
G[llama.cpp master CUDA sm_86
|
||
frank-secondary, port 7771]
|
||
H[Swift-1.5-27B Uncensored RCO-IQ3_XXS-mtp GGUF
|
||
2 x 131072 ctx, q8_0 K + q8_0 V, draft-mtp depth 3, mmproj vision]
|
||
E2 --> E --> F
|
||
E2 --> G --> H
|
||
end
|
||
|
||
CLI[LLM Clients / API] -->|HTTP :7770| B2
|
||
CLI -->|HTTP :7770| E
|
||
CLI -->|HTTP :7771| G
|
||
```
|
||
|
||
## Servers
|
||
|
||
| Server | GPU | VRAM / Memory | OS |
|
||
| --------------------------------------- | ----------------------------- | -------------------- | ------------ |
|
||
| [[Dankserver System Specs\|Dankserver]] | 2x AMD Radeon AI PRO R9700 | 2x 32 GB VRAM, 246 GB RAM | Ubuntu 26.04 |
|
||
| [[Dankclaw System Specs\|Dankclaw]] | AMD Radeon 8060S (Strix Halo) | gfx1151 unified memory | Ubuntu 26.04 |
|
||
|
||
## Active Deployment
|
||
|
||
- [[frank-large-flash-server-v2-swift15]] — **ACTIVE since 2026-10-06**
|
||
(root systemd unit + podman, port 7770): the digest-pinned
|
||
`ghcr.io/peonist-ai/halogen-flash-server` 0.16.2 engine serving the
|
||
Quat3rnion ABLITERATED Swift1.5 finetune of peonist's
|
||
Qwen3.8-Flash-Next v2 checkpoint (`qwen38-flash-next-v2-swift15-abliterated.hgn`,
|
||
66.7 GB compacted r2 of the card's 89.8 GB append-only original,
|
||
sha256-verified) with the REQUIRED n-gram lookup table (51.2 GB,
|
||
@peonist 5cc17cea) and the vision tower (897.9 MB, @e053f488) beside
|
||
it. TWO slots of 200,192 ctx, MTP head ALWAYS ON (`HALOGEN_SPEC_ADAPT=0`
|
||
— the adaptive default turns the head off per request and halves
|
||
decode), server-side sampler at the card's 1.0/0.95/20, coupled
|
||
cap/timeout 65536/7200, prompt cache ON by default (the 27B engine's
|
||
manual `HALOGEN_CACHE_MB` pin era is over — warm 100k turn ≈ 2 s vs
|
||
≈ 88 s cold). Vision ON (image_url parts only). Card: 53-57 t/s
|
||
greedy, ~1,640 t/s prefill at 34k, ~72 % fewer thinking tokens.
|
||
Rollback: revert the swap commit; the 27B payload + fetch script stay
|
||
on disk
|
||
- [[frank-large-halogen27b-dflash2]] — **RETIRED 2026-10-05 → 10-06
|
||
(rollback target; payload + fetch script + image on disk)**: the
|
||
digest-pinned `ghcr.io/peonist-ai/halogen` 0.1.4 engine serving
|
||
`peonist-ai/halogen-qwen3.8-27b` (`qwen3.8-27b-p1w4d-d2.hgn`, 33.4 GiB)
|
||
— DFlash2 drafter, ONE 200,700 slot, text-only (the engine ignored the
|
||
vision encoder), censored base. Owns the measured lessons: the prompt
|
||
cache pin (`HALOGEN_CACHE_MB=16384` — auto was OFF at the 36.9 GB
|
||
two-entry floor; a 17.3k cold prefill of 27.0 s resumed in 1.7 s warm),
|
||
the FA tile rejection (`HALOGEN_FA_TQ=256`: prefill −9.5 %, decode
|
||
±0.3 %), and deep decode at 3.3-3.8 t/s@80k being structural. The
|
||
foreground-`podman run` ledger capture and the deep-role rerouting
|
||
(`slow`/`advisor` → frank-primary) carried forward. Superseded by the
|
||
MODEL swap: the abliterated v2 .hgn loads only in flash-server, so the
|
||
engine moved with the model
|
||
- [[frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688]] — **ACTIVE since
|
||
2026-10-05 (evening); UNCECERSED RCO-IQ3_XXS with MTP back on since
|
||
2026-10-06 evening; RENAMED from frank-small and reshaped to 2 ×
|
||
114,688 / ubatch ladder 512 → 1024 → **2048** late 2026-10-06** (root
|
||
systemd unit, port
|
||
7771): llama.cpp master
|
||
(CUDA sm_86) serving ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-
|
||
GGUF **RCO-IQ3_XXS-mtp** (27B; the MTP head in the file is DRAFTED —
|
||
draft-mtp depth 3, ngram skipped on the recurrent-rollback law) on the
|
||
RTX 3090, TWO 114,688-token conversations at Frank's 112k cap (pool
|
||
229376 / parallel 2 — cut from 131,072 to fund the ubatch rungs;
|
||
**1,005 MiB free at boot, 763 settled after a vision probe**) with
|
||
q8_0 K + q8_0 V (FP8 class, 34 KiB/token vs 64 at f16) and
|
||
vision through the publisher's own mmproj-BF16.gguf. Prefill 1,209
|
||
t/s @ 3,760; decode ~52–58 t/s recipe (ubatch-invariant; acceptance
|
||
noise 34–54 %).
|
||
RCO-IQ3_XXS tier is a per-tensor allocation mix of stock types; the
|
||
payload is the UNCECERSED Swift 1.5 (23/100 refusals vs Swift 1.5's
|
||
98/100). Replaces BOTH same-day payloads: the live MiMo Q6_K shape
|
||
(below) and the Ornith-1.5-9B Q8_0 swap that was committed
|
||
(77decb5..380aafa) but never deployed
|
||
- [[frank-secondary-llama-mimo-q6k-2x200704]] — **RETIRED 2026-10-05 (evening;
|
||
rollback target; payload on disk)**: llama.cpp master (CUDA sm_86)
|
||
serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, TWO slots of
|
||
200,704 with FULLY UNQUANTIZED KV (f16 K + f16 V) and the mmproj for
|
||
vision — 94.4 tok/s decode (the rung costs a measured -10 % vs Q5_K_M:
|
||
weight-bandwidth-bound), batch 4096 / ubatch 2048 since 2026-10-05
|
||
(+4.6 % prefill, measured), 2,279 MiB of the card left free. This was
|
||
the LIVE payload when Swift cut over. The ubatch A/B table and the
|
||
probe-size trap (>=256 px images or the model says "video") live inside
|
||
- [[frank-secondary-llama-mimo-q5km]] — **RETIRED 2026-10-02 → 10-03 (rollback
|
||
target; payload on disk; owns the measured boot-pair table)**: the same
|
||
runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower —
|
||
104.8 tok/s, 3,571 MiB true free
|
||
- [[frank-primary-radiance-qwen38-native-fp8]] — **ACTIVE since 2026-10-05**
|
||
(root systemd unit, port 7770): the StillDeadcode **radiance** engine
|
||
(C++/HIP, libr4d gfx1201 kernels, digest-pinned 287 MiB image, engine as
|
||
the image entrypoint) serving the OFFICIAL `Qwen/Qwen3.8-Flash-Next-FP8`
|
||
checkpoint at NATIVE precision — converted no-recipe to
|
||
`qwen4exp-fp8-native.rad` (~161 GiB) after dropping `quantization_config`
|
||
in a staging dir (the plugin descriptor-selection law: nothing claims
|
||
`(qwen4exp, "fp8")`, the shipped plugin claims `(qwen4exp, "")`;
|
||
per-weight encodings are data, not the descriptor) — on BOTH R9700s
|
||
**TP=2** (`--tp-wire wht6` fused all-reduce), experts in a 40 GiB/rank
|
||
pinned host pool + VRAM cache (ratio 0.82 = the measured 99 %-hit point),
|
||
the 47.7 GiB PLE n-gram table PINNED to host RAM (`--ngram-placement
|
||
ram`), fp8 KV 7.32 KiB/token, MTP spec decode depth 3, 2 × 200,000 first
|
||
boot, froggeric v22.5 template, vision + tools + `reasoning_content` auto,
|
||
`/metrics` scraped. Rollback: git tag
|
||
`frank-primary-freetoken-pr587-20261005` (FreeToken image + orcarouter
|
||
payload on disk); older rollback `frank-primary-vllm-shapeG-20261003`
|
||
- [[frank-primary-freetoken-pr587]] — **RETIRED 2026-10-05 (same day;
|
||
rollback target)**: FreeToken at PR #587 head `77a7201` (branch
|
||
rocm714_torch214) built on the host into the local image
|
||
`frank-primary-freetoken:pr587-77a7201` (Dockerfile: digest-pinned
|
||
rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev-symlink law,
|
||
curated --no-deps install per docs/install_amd.md), serving the SAME
|
||
orcarouter FP8 payload on ONE R9700 — experts in pinned host RAM
|
||
(~120 GiB + 47.7 GiB PLE), GPU LRU at 0.10, KV pool 299,968 tokens
|
||
(multiple-of-64 law), 2 slots, parsers AUTO (thinking →
|
||
`reasoning_content`). Measured: boot ~2.3 min, decode 17.1 tok/s
|
||
single-stream, vision "Red" probe passes, native tool_calls; TP=2 does
|
||
not boot on ROCm (pynccl JIT) — second card idle. Superseded by the
|
||
radiance swap for TP=2; the image + payload stay on disk until the
|
||
rollback window closes
|
||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **RETIRED 2026-10-05
|
||
(rollback target; the vLLM trio + env lore live in the tag)**: the
|
||
tcclaviger RDNA4
|
||
vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both
|
||
R9700s, TP=2 with automatic expert offload — `--max-model-len 200704` x 3
|
||
slots, fp8 KV, pool 3,150,000,000 B/rank (measured 398,658 tokens = 1.99x),
|
||
MNBT 1024, gather 32x64, no spec decode, prefetch off, plus
|
||
`--expert-precise-memory` (arena 17.22 GiB / 7,522 slots), GPU_MAX_HW_QUEUES=2
|
||
and a mounted compile cache. Why `--max-num-seqs`
|
||
must stay 3 (SEQS 2 cannot compile this model), why the arena's levers are
|
||
`max_model_len` and the overhead term rather than the KV pool, and the measured envelope
|
||
- [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
|
||
failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro):
|
||
the tcclaviger RDNA4 vLLM fork
|
||
(digest-pinned `tcclaviger/vllm@sha256:da4029666…`, plain `podman run`,
|
||
no shim) serving Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric
|
||
int4 group 128, ~170 GiB, 34 shards + 2 extras, 113,074 tensors — on BOTH
|
||
R9700s, TP=2, with the MoE experts offloaded to pinned host RAM (12,096
|
||
experts/rank, 73.64 GiB) behind a 14.51 GiB / 2,383-slot VRAM LRU arena; the
|
||
MoE kernel is libr4d's grouped W4A16 GEMM (`R4dW4a16MoEMethod`, reached
|
||
because the checkpoint's `quant_method: auto-round` is overridden to `inc` —
|
||
a compressed-tensors W4A16 checkpoint of the same model has NO offload recipe
|
||
and refuses to serve). 131,072 ctx x 2 seqs, bf16 KV, util 0.98, prefix
|
||
caching, vision and `glm47` tool/reasoning parsers, MTP OFF and prefetch OFF
|
||
by measurement. **Measured decode 9.9-10.0 tok/s, prefill 918 tok/s — the
|
||
35 tok/s target is NOT met, bounded by the fork's expert-gather read path
|
||
(~8.2 GB/s effective vs a 22.74 GB/s link) at a 0.53-0.66 cache hit rate.**
|
||
Rollback: commit `2339fae` (the qwen FP8 + MTP4 shape of 2026-09-29 → 09-30)
|
||
- [[frank-primary-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
|
||
back; payload and R9V state dirs intact on disk, git tag
|
||
frank-primary-r9v-q8-20260929 is the last pre-cutover commit):
|
||
[[frank-primary-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
|
||
(same-day swap to the Q8_0 payload; rollback target, state + payload
|
||
intact on disk): the R9V radiance-vLLM fork serving
|
||
Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) +
|
||
blockfp8 MTP4 (depth 4) + F16 vision on BOTH R9700s — TP=2 tiered
|
||
experts (UVA pinned host copy, 62 hot + 160-slot LRU on rank 0, 428
|
||
static on rank 1), PLE n-gram on ssd-mmap, CED on, 196608 ctx. Cut over
|
||
from llama.cpp the same day; rollback target
|
||
[[frank-primary-llamacpp-qwen38-flash-next-q8]]
|
||
- [[frank-primary-llamacpp-qwen38-flash-next-q8]] — RETIRED 2026-09-27
|
||
(rollback target; the llama.cpp b11030 endpoint of 2026-09-26):
|
||
unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X, bundled runtime)
|
||
serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 across both R9700s —
|
||
routed experts and the 51B-param n-gram table computed in host RAM
|
||
(`-ot exps=CPU,per_layer_token_embd.weight=CPU`), the shared Q8_0 MTP head
|
||
(depth 2) chained behind ngram-mod, BF16 vision, one full 262,144-token
|
||
conversation, `reasoning_content`, the placement law, the cutover and
|
||
rollback
|
||
- [[frank-primary-lucebox-ds4]] — RETIRED 2026-09-26 (same-day swap,
|
||
rollback target; active for hours): the Lucebox DS4.1 engine
|
||
(localhost/lucebox-rocm:gfx1201) serving
|
||
Lucebox/DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB) with streamed experts —
|
||
the fleet's only non-llama.cpp runtime detour
|
||
- [[dankserver-frontier-moe-feasibility]] — VERIFIED 2026-09-28: which of
|
||
DeepSeek-V4.1-Flash (552B), MiMo-V2.6-Flash-MOPD (309B) and GLM-5.3-Flash
|
||
(320B) can run on the 2x R9700 box, on what runtime, and why V4.1 does not
|
||
fit (475 GiB payload, 189 GiB of it Engram; decode is PCIe-expert-stream
|
||
bound, ~23 tok/s ceiling at today's residency). Includes the Affinity
|
||
V4-Flash-to-V4.1 port assessment — reusable expert tiering, four
|
||
architecture gaps, one of them a silent-wrongness trap on the Engram key
|
||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — VERIFIED 2026-09-29:
|
||
the flag-level decode-tuning campaign on the ad hoc qwen-fp8 container
|
||
(tcclaviger RDNA4 fork, orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8,
|
||
TP=2 across both R9700s). **>35 tok/s MET: 37.44-40.57 tok/s with
|
||
`--expert-prefetch` DELETED** — telemetry measured the prefetcher at
|
||
36% precision, wasting ~half the gather traffic (the LRU itself runs at
|
||
82.45% hit). Prefetch-on incumbent 26.8 survived every other challenger
|
||
(geometry, spec depth, TunableOp-corrupts, batch knobs, concurrency,
|
||
ngram-fork-bug); the residual gap to ~60 tok/s is the fork's gather
|
||
read path (6.5-9.5 GB/s effective vs 22.7 at the same slab size)
|
||
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26 (replaced by
|
||
[[frank-primary-lucebox-ds4]], then
|
||
[[frank-primary-llamacpp-qwen38-flash-next-q8]]): vllm-radiance 0.9.3
|
||
serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark
|
||
W4A8) with the z-lab DFlash2-FP8 drafter as ONE tensor-parallel process
|
||
(TP=2) behind :7770, on unquantized bf16 KV (`KV_DTYPE=auto`, cut
|
||
2026-09-21). The FP8 checkpoint is the on-disk rollback. The three-arm
|
||
battery that chose TP=2, the auto-sized KV pool, and the verified
|
||
capability battery (tools, reasoning)
|
||
- [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium:
|
||
the streamlining toolchain (hf-manifest.py, hf-fetch.sh,
|
||
swap-frank-medium.sh), the quantization-aware entrypoint contract, the
|
||
quadlet edit, the deploy, the capability battery, and the rollback path.
|
||
The 2026-09-20 MXFP4 → FP8 swap is the worked example
|
||
- [[frank-medium-llamacpp-dflash2]] — RETIRED 2026-09-16 → 2026-09-20: mainline
|
||
llama.cpp serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the
|
||
DFlash2 Q4_K_M drafter as ONE layer-split process (32:33 over two
|
||
163,840-token slots); superseded by the radiance TP=2 cutover, and the
|
||
rollback target. Keeps the batch/ubatch optimum curve and the per-layer split
|
||
arithmetic
|
||
- [[Dual-R9700-Cooperation]] — whether the two R9700s can serve ONE model: the
|
||
RCCL/PCIe-atomics wall on the old TUF GAMING B650-E, the 2026-09-20 ProArt
|
||
B850-Creator x8/x8 swap that cleared it, and the measured three-way verdict
|
||
(radiance TP=2 95.8 t/s decode vs layer split 45.7 vs llama tensor mode 30.5)
|
||
that put TP=2 in production
|
||
- [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era,
|
||
2026-09-15): charlie12345/ROCmFPX serving
|
||
kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the
|
||
runtime it replaced
|
||
- [[freetoken-frank-small]] — RETIRED 2026-09-15, when the RTX 3090 moved to
|
||
frank-large: the FreeToken v0.1.2 frank-secondary runbook (batsclamp
|
||
Huihui-Qwen3.6-35B-A3B-abliterated-FP8), kept as the retirement record. The
|
||
NAME was reused on 2026-09-16 by a different service on Dankclaw — see
|
||
[[frank-secondary-qwen38-gsq-rco]]
|
||
- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep
|
||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||
- [[deployment]] — deploy and update process for both servers
|
||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||
- [[frank-large-rocmfpx-q6agent-uncensored-mtp]] — **RETIRED 2026-10-03 →
|
||
10-05 (rollback target; payload, fork build and build script on disk)**:
|
||
the pinned ROCmFPX fork serving the uncensored protected-heads
|
||
Q6_0_ROCMFPX_AGENT rung with the repo's MTP-Q4_0 head drafting 4 tokens,
|
||
one 262,144 window, f16 KV; measured 27.6-28.1 t/s prose decode at 86.8 %
|
||
acceptance. Replaced by the halogen DFlash2 endpoint; the ngram-mod-onto-MTP
|
||
rejection (-15 %, recurrent-state rollback) stays a live lesson for this
|
||
architecture.
|
||
- [[frank-large-rocmfpx-q8agent-mtp]] — **RETIRED 2026-10-02 → 10-03 (the
|
||
rollback target; payload still on disk)**: the same pinned ROCmFPX fork
|
||
serving the ALIGNED `Q8_0_ROCMFPX_AGENT` (ftype 115, 17.0 t/s, MTP
|
||
acceptance 0.462) — the MTP-vs-none +123 % A/B, the trap catalogue
|
||
- [[frank-large-gufo-qwen38-27b]] — **RETIRED 2026-10-02 (history; the
|
||
payload dirs were removed from disk 2026-10-03, so this rollback is a
|
||
re-download)** (root systemd unit, port 7770): gufo
|
||
(gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from
|
||
source on Dankclaw, serving unsloth/Qwen3.8-27B-GGUF **UD-Q4_K_XL**
|
||
with the z-lab DFlash2 Q4_K_M drafter — ONE session at the full native
|
||
262,144 window, unquantized f16 KV. Rolled back by Frank after one day
|
||
of [[frank-large-vllm-mimo-v26-bf16]]. The two from-source build
|
||
workarounds, the client-contract delta, the measured prefill/decode
|
||
trade against llama.cpp
|
||
- [[frank-large-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (one day;
|
||
rollback by Frank after the autoresearch loop): the digest-pinned
|
||
kyuz0/vllm-therock-gfx1151 image serving
|
||
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16, 4 x 262144 ctx,
|
||
CROSS-MODEL MTP from Qwen/Qwen3.5-9B — kept for the measured record:
|
||
battery decode 6.19 t/s (MTP-2) vs 9.01 t/s (no draft — spec NET-
|
||
NEGATIVE on real chat+reasoning traffic), the roof math, the greedy-text
|
||
numerics trap, and the death of every ≥6-bit vLLM quant path on gfx1151
|
||
(quark int8 rejected, FP8 kernels CDNA-only, torchao unwired)
|
||
- [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
|
||
[[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp
|
||
(unsloth b11030, ROCm gfx1151) serving
|
||
unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2
|
||
Q4_K_M drafter (depth 4) on the gfx1151 iGPU — one full 262,144-token
|
||
conversation, q8_0 KV (the build lacks fp8), text-only, Qwen-Sharp
|
||
v22.4.0 template, the 114 GiB GTT memory model, client contract,
|
||
troubleshooting
|
||
- [[frank-large-kairic-edge-qwen38-27b]] — RETIRED 2026-09-21 (same-day
|
||
swap, replaced by [[frank-large-unsloth-qwen38-27b-q8xl]]; rollback
|
||
target): the Kairic Edge runtime (ciru-ai/ROCmFPX v1.2, built from
|
||
source) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (IU4 W4A4, 27B dense)
|
||
on the gfx1151 iGPU — one full 262,144-token conversation, MTP depth 4 +
|
||
n-gram drafting, the three PromptForge sidecars, compatibility mode, the
|
||
114 GiB GTT memory model, client contract, troubleshooting
|
||
- [[frank-large-unsloth-qwen38-flash-next]] — RETIRED 2026-09-21 (same-day
|
||
swap, replaced by [[frank-large-kairic-edge-qwen38-27b]]; rollback target):
|
||
the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving
|
||
unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (125B MoE) on the gfx1151 iGPU —
|
||
one 200,000-token conversation, MTP depth 2 via the explicit -md shared-Q8_0
|
||
head, the 114 GiB GTT memory model, client contract, troubleshooting
|
||
- [[frank-large-ciru-orca]] — RETIRED 2026-09-21 (replaced by
|
||
[[frank-large-unsloth-qwen38-flash-next]]; rollback target): the CIRU
|
||
v4.4.1 runtime (private host build, QSA hotfix) serving
|
||
jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU — one
|
||
262,144-token conversation, MTP depth 4, 4096 MiB PLE cache, runtime-root
|
||
gate, client contract, troubleshooting (2026-09-19 → 09-21)
|
||
- [[halogen-flash-server]] — RETIRED 2026-09-19 (replaced by
|
||
[[frank-large-ciru-orca]]; rollback target): peonist-ai halogen-flash-server
|
||
0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b
|
||
.hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
|
||
400,384-position pool, deployed flags, memory model, client contract,
|
||
troubleshooting (2026-09-16 → 09-19)
|
||
- [[frank-secondary-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02** (the same
|
||
rung returned ACTIVE 2026-10-03 at the 2 x 200,704 f16-KV shape —
|
||
[[frank-secondary-llama-mimo-q6k-2x200704]]): llama.cpp (master, CUDA sm_86
|
||
build) serving
|
||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO
|
||
native 262,144-token sessions with ASYMMETRIC KV (K f16 unquantized + V
|
||
q8_0, which forces flash-attn), ~20.9/24.6 GiB measured, text-only (the
|
||
GGUF has no vision), reasoning_content, no spec decode; payload still at
|
||
`/home/dank/models/MiMo-Q6K/`
|
||
- [[frank-secondary-sglang-mimo-v26]] — RETIRED 2026-10-01 (hours; replaced
|
||
by [[frank-secondary-llama-mimo-q6k]] for the asymmetric KV SGLang cannot
|
||
express; venv still on disk, INT8 payload staged to
|
||
`~dank/.AGENTTRASH/20261002-retired-llm-payloads/`): SGLang in a
|
||
self-contained python3.12 venv serving leteschlead MiMo-V2.6 INT8
|
||
W8A16 AutoRound, 2 x 262144 ctx, fp8 KV, in-checkpoint vision,
|
||
83.9 tok/s decode
|
||
- [[frank-secondary-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by
|
||
[[frank-secondary-sglang-mimo-v26]] → the llama.cpp MiMo shape; payload staged
|
||
to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/`):
|
||
mainline llama.cpp serving
|
||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
|
||
3090, with the separate MTP assistant sidecar chained behind ngram-mod,
|
||
q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB.
|
||
- [[frank-secondary-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
|
||
rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
|
||
Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-
|
||
token pool, 2 concurrent full-context sessions, native text+image+audio.
|
||
- [[frank-secondary-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the
|
||
prior frank-secondary runbook (2026-09-16 → 09-17), ISTA-DASLab
|
||
Qwen3.8-27B-GSQ-RCO IQ3_S with the model's own in-file MTP head and
|
||
f16 KV at 131k
|
||
- [[x4-pair-experiment]] — MEASURED 2026-09-16: what mainline llama.cpp can and
|
||
cannot do with the 3090 + Strix Halo pair, over their only link (the 3090's
|
||
Gen4 x4). The tensor inventory that decides any split, the single flag (`-ot`)
|
||
that kills per-tensor placement across devices, the two-device layer splits
|
||
that work (+7 % decode at 64k, cross-vendor included), the harness-parsing trap
|
||
that invalidated the first round, and the iGPU-only result that matches
|
||
halogen's deployed decode rate — at a memory cost the co-tenants cannot pay
|
||
- [[frank-large-ik-uncensored]] — RETIRED 2026-09-16 (rollback target): the
|
||
ik_llama.cpp runbook (root systemd unit, 2026-09-15 → 09-16) serving
|
||
mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090
|
||
(dense/attention/KV) + LPDDR (experts/PLE) hybrid; its measured envelope and
|
||
decode root-cause chain are the record of why the fleet went back to halogen
|
||
- [[dankclaw-iommu-translation-off]] — why Dankclaw boots with
|
||
`amd_iommu=off` since 2026-09-11 (13-16 % prefill, measured upstream),
|
||
what it cost (the NPU endpoint), what was deliberately not copied from
|
||
the reference machine, and how to revert
|
||
- [[kingjones-strix-rocmfp4]] — PRIOR frank-large runbook and rollback
|
||
target: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params
|
||
(56B MoE + 51.2B PLE), single 113.47 GiB GGUF, on the card's pinned
|
||
kingjones30/ROCmFPX fork (2026-08-30 → 2026-09-11)
|
||
- [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
|
||
- [[frank-medium-batch-ubatch]] — batch/ubatch sweep on the 2x R9700 layer split
|
||
(2026-09-16): the ubatch optimum at 512, why the logical batch is free, and
|
||
the split rebalance that followed it
|
||
- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
|
||
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model
|
||
on Dankclaw (retired)
|
||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on
|
||
Dankserver (frank-secondary, retired 2026-09-02)
|
||
- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for
|
||
frank-secondary (2026-08-26): q8_0 + Hadamard KV cache,
|
||
split-mode none (single-GPU), mandatory FA_ALL_QUANTS build flags
|
||
- [[kingjones-strix-mtp]] — the former Q6 AGENT runbook (restored 2026-09-01
|
||
after Q8_0 failure, then superseded by the STRIX deployment on 2026-08-27)
|
||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime
|
||
on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
|
||
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
|
||
(deployed and reverted 2026-08-22)
|
||
- [[Dankserver System Specs]] — the 2x R9700 box (frank-primary)
|
||
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on
|
||
the flash-server v2 runtime since 2026-10-06, frank-secondary on the RTX 3090)
|
||
- [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint
|
||
on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as
|
||
the retirement record — the IOMMU/SVA err=-19 root cause and where
|
||
every dependent route moved
|
||
|
||
## CIRU Reference (Dankclaw)
|
||
|
||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 model, ROCm 7.15 toolchain,
|
||
and build history (on Dankclaw; the slot now serves the kingjones
|
||
ROCmFP4 model, see [[kingjones-strix-mtp]])
|
||
- [[kairic-edge]] — Kairic Edge IU4 runtime: build runbook, binary
|
||
identity, and experiment summary (retired 2026-08-22)
|
||
|
||
## DFlash2 Reference (archived)
|
||
|
||
- [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment
|
||
(2026-08-21)
|
||
|
||
## vLLM Reference (historical)
|
||
|
||
- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
|
||
- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
|
||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware
|
||
notes and sizing (this architecture family is what the retired frank-secondary
|
||
served in the Huihui abliterated FP8 variant — see
|
||
[[freetoken-frank-small]]; the Ornith-1.5-35B-A3B CRACK checkpoint
|
||
that preceded it is archived at [[dealign-ornith-crack]])
|
||
|
||
## Memory Management
|
||
|
||
- [[RAM CPU Offloading in VLLM]] — KV cache swap between GPU VRAM and system RAM
|
||
- [[Hybrid CPU GPU Processing Dankserver]] — NVIDIA GPU compute with system RAM offload
|
||
- [[Shared Memory Architecture Dankclaw]] — AMD unified memory and ROCm configuration
|
||
|
||
## Retired Topology (2026-08-17 redo)
|
||
|
||
- [[Qwen3.8 PP Deployment]] — the retired pipeline-parallel head + worker
|
||
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history (superseded by FastMTP)
|
||
- [[FastFlowLM NPU Inference]] — the retired FastFlowLM NPU endpoint
|
||
- [[frank-large-benchmark]] — backend/spec benchmark behind the old head
|
||
- [[dankserver-vram-growth]] — VRAM/RAM growth behind the retired cuBLAS crash
|
||
- [[DFlash DSpark Speculative Decoding]] — retired DFlash/DSpark drafting
|
||
- [[Quantization by Hardware Architecture]] — which quant format fits which GPU/CPU
|
||
|
||
## Pipeline Parallelism (archive)
|
||
|
||
- [[Pipeline Parallelism/index]] — index of multi-node inference research. The PP
|
||
pair is retired. These pages are research history.
|
||
|
||
## Runtime Research (llama.cpp family)
|
||
|
||
- [[Llama Cpp Runtimes]] — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: positioning, quantization, server fit
|
||
- [[Llama Cpp Code Diff]] — code-level quantization diff of the three runtimes + server recommendations
|
||
- [[SGLang]] — the structured-generation serving engine; when it fits Dankserver (2x R9700) vs Dankclaw (Strix Halo)
|
||
|
||
## Cross-Cutting References (skill distillation, 2026-10-05)
|
||
|
||
One polished reference page per skill-cluster, distilled from the managed
|
||
skill library into the vault (see [log](../log.md) for the wave plan):
|
||
|
||
- [[GGUF Compatibility]] — what a GGUF's tensor histogram and header demand
|
||
of a loader; how the runtime is chosen by the file
|
||
- [[Llama Cpp Serving Sizing]] — slot/pool/KV sizing laws, boot-shape
|
||
laddering, llama-bench discipline
|
||
- [[Llama Cpp Multi-Device Splits]] — layer vs tensor splits, `-ts`/`-ot`
|
||
laws, drafter placement
|
||
- [[PCIe Link and Atomics]] — the link walls, atomics routing, and the
|
||
heterogeneous-pair verdicts
|
||
- [[RDNA4 Container Serving]] — bring-up recipes for RDNA4 containers
|
||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan law, every arena
|
||
lever, and why free VRAM is not arena
|
||
- [[vLLM Boot Forensics and Config Interrogation]] — reading a boot that
|
||
died, and interrogating a running fork's config
|
||
- [[vLLM Porting and Compile Triage]] — porting checkpoints across vLLM
|
||
versions/forks; torch.compile/dynamo breakage triage
|
||
- [[LLM Bench Windows and Measurement]] — window discipline, honest A/B,
|
||
the window script laws
|
||
- [[LLM Performance Diagnosis]] — slow prefill/decode triage on live
|
||
endpoints, IBS attribution, prefix-cache routing
|
||
- [[Model Swap and Rollback]] — the swap procedure, payload preflight,
|
||
research verification, second-endpoint bring-up
|
||
- [[Tcclaviger Fork Ops]] — the fork's sweep harness, measured knob
|
||
verdicts, large-MoE recipe, compile-cache persistence
|
||
- [[Heterogeneous Split Triage]] — triage for unequal-accelerator split
|
||
proposals (the no-go verdict format)
|
||
- [[FreeToken Builds]] — the CUDA and ROCm build paths with their
|
||
dependency traps
|
||
|
||
These sit alongside the operational pages under
|
||
`../Operations/index.md` and `../Agent/index.md`.
|
||
|
||
## Related
|
||
|
||
- [Project knowledge base](../index.md) — the Podman app stacks on the same
|
||
two hosts. OpenWebUI (Dankclaw) consumes the on-host `frank-large`
|
||
endpoint at `:7770`. The Gitea instance on Dankclaw hosts this repository.
|
||
- [Deployment Flow](../Architecture/Deployment%20Flow.md) — the service-scoped
|
||
git-based deploy with inference isolation |