docs(vault): one page per endpoint, forever -- 45 era pages fold into three
Every payload swap a live endpoint has ever run is now one row in that endpoint's Eras table, with its rollback path in the same row. No per-attempt page, no per-swap ledger, nowhere else. - Added: LLM/frank-archon.md (radiance, 2x R9700 TP=2), LLM/frank-dragoon.md (halogen flash-server, 19 eras), LLM/frank-zealot.md (llama.cpp PrismML fork on the 3090). Each carries Current shape, Eras, Laws, wire contract, files. - Deleted: 45 per-payload runbooks -- the frank-dragoon line (11 including the pre-name halogen/kingjones/kairic/CIRU/eemin/agentionai/dflash2 pages and the benchmark page), the frank-zealot line (llama.cpp/vLLM/ik_llama/SGLang eras), the frank-archon line (vLLM/FreeToken/R9V/llama.cpp/radiance eras), plus frank-medium, frank-npu and the ubatch/PCIe one-offs that owned nothing. - Rewired: every wikilink and markdown-path reference to a folded page points at the endpoint page instead; the three Research see-also lists and the ubatch sweep's runbook link included. No anchored links existed in the set. - Regenerated both index pages as pure locators -- one row, one clause, no numbers -- so the docs_completeness locator law holds for all 153 pages. tests/docs_completeness.bats green: index links every page, no unresolved links. The pages moved to .AGENTTRASH/20261018-doc-dedupe/LLM/ (gitignored, and git history holds every byte).
This commit is contained in:
Frank
@@ -1,647 +1,89 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
# LLM fleet — index
|
||||
|
||||
**Dankserver runs the radiance engine (StillDeadcode) on frank-archon**
|
||||
(digest-pinned `stilldeadcode/radiance` image, engine as entrypoint,
|
||||
driving `podman run` directly, root systemd unit, :7770):
|
||||
the **publisher's** 4-bit-codebook container of the official
|
||||
`Qwen/Qwen3.8-Flash-Next` (`qwen3.8-next-flash-fp8-iq4r-moe.rad`, 116,322 MiB,
|
||||
sha256-verified at fetch — w4nl codebook experts + int8 trunk, PLE n-gram,
|
||||
MTP head and vision tower in-container), on BOTH R9700s
|
||||
TP=2 (`--tp-wire wht6`), experts in a 44 GiB/rank pinned host pool + VRAM cache
|
||||
(explicit `--vram-weights-mib 25600 --vram-kv-mib 5120` = 30,720 of the
|
||||
30,969 MiB the engine's own budget table makes claimable), the 47.7 GiB PLE
|
||||
table pinned to host RAM, **fp8 KV at 3 × 200,000**, MTP spec decode depth 2,
|
||||
froggeric v22.5 template. Rollback: revert the swap commit and re-download the
|
||||
in-house Swift1.5 w4dn8 `.rad` byte-exact (purged from disk 2026-10-17; Hub
|
||||
`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`); tags
|
||||
frank-archon-freetoken-pr587-20261005 and frank-archon-vllm-shapeG-20261003
|
||||
(AGENTS.md Runbook Index; `find /home/dank/models -name '*.rad'` on
|
||||
Dankserver 2026-10-18 lists only the iq4r file),
|
||||
and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2
|
||||
(frank-dragoon, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned
|
||||
digest serving the NATIVE v2 conversion of the abliterated
|
||||
Qwen3.8-Flash-Next, a ~62 GiB v2 .hgn that loads in no other engine,
|
||||
MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots over ONE
|
||||
401,408-position KV pool — 2 × ctx, raised and DEPLOYED 2026-10-18 — at ctx
|
||||
200704, vision ON)** and
|
||||
**llama.cpp on the PrismML fork (frank-zealot, port 7771: prism-ml
|
||||
Ternary-Bonsai-2-9B at **PQ2_0** — 9B qwen35 hybrid (8 full-attention + 24
|
||||
Gated DeltaNet layers + the blk.32 MTP block), TWO slots over ONE
|
||||
524,288-position pool (2 × the 262,144 window), MTP present in the file but
|
||||
OFF by measurement, **exact f16 K + f16 V** — 32,768 B/token = **16,384 MiB**
|
||||
of KV (three 262,144 windows at exact KV do not fit the RTX 3090), vision via
|
||||
the publisher's own Q8_0 mmproj, on the RTX 3090 — and the custom
|
||||
tensor type 141 + `prism.hadamard.*` metadata mean no other loader can open the
|
||||
file)**.
|
||||
The three live endpoint pages are the only pages that carry a shape. Everything
|
||||
else here is a technique, a piece of hardware, or a trial that taught a law — and
|
||||
each of those owns exactly that one thing. The incidents behind the laws are in
|
||||
[[log]].
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
> `serverAppConfigs-docs/` root to `serverAppConfigs-docs/LLM/`. The fleet scripts and units now live in the
|
||||
> repo's `scripts/` directory.
|
||||
## Live endpoints
|
||||
|
||||
## Deployment Topology
|
||||
| Page | Owns |
|
||||
|---|---|
|
||||
| [[frank-archon]] | The one frank-archon page: the live radiance shape and harness contract (read off scripts/frank-archon.env), the Eras… |
|
||||
| [[frank-dragoon]] | The one runbook for frank-dragoon, the Strix Halo iGPU endpoint (192.168.0.143:7770): current shape from the env,… |
|
||||
| [[frank-zealot]] | The one frank-zealot reference: the live shape on Dankclaw's RTX 3090 (:7771) exactly as scripts/frank-zealot.env… |
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
|
||||
A[2x Radeon AI PRO R9700
|
||||
gfx1201, 32 GiB each]
|
||||
B2[radiance StillDeadcode, digest-pinned image
|
||||
podman run, frank-archon, port 7770]
|
||||
C[Publisher qwen3.8-next-flash-fp8-iq4r-moe .rad
|
||||
2x R9700 TP=2 wht6, experts 44 GiB/rank pinned pool + VRAM,
|
||||
PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 2,
|
||||
3 x 200000, froggeric v22.5]
|
||||
end
|
||||
## Retired endpoints
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
E2[RTX 3090 (PCIe x4 dock) +
|
||||
gfx1151 iGPU 114 GiB GTT]
|
||||
E[flash-server container gfx1151, digest-pinned image
|
||||
frank-dragoon, port 7770]
|
||||
F[flash-server v2 native abliterated .hgn + ngram table + vision tower
|
||||
ONE 401408-position KV pool at ctx 200704 = 2 slots x ctx, MTP always-on, image input]
|
||||
G[llama.cpp PrismML fork CUDA build
|
||||
frank-zealot, port 7771]
|
||||
H[Ternary-Bonsai-2-9B PQ2_0 GGUF + Q8_0 mmproj
|
||||
2 slots over ONE 524288-position pool at ctx 262144, exact f16 K + f16 V, MTP off, mmproj vision]
|
||||
E2 --> E --> F
|
||||
E2 --> G --> H
|
||||
end
|
||||
| Page | Owns |
|
||||
|---|---|
|
||||
| [[frank-medium-batch-ubatch]] | llama-bench sweep of --batch-size against --ubatch-size on frank-medium (2x Radeon AI PRO R9700, layer split) at the… |
|
||||
| [[frank-medium-llamacpp-dflash2]] | The former ACTIVE frank-medium deployment since 2026-09-16 (model swapped 2026-09-19) — mainline llama.cpp on… |
|
||||
| [[frank-medium-model-swap]] | How to swap the checkpoint behind the frank-medium endpoint (Dankserver :7770, vllm-radiance TP=2) — the streamlining… |
|
||||
| [[frank-medium-radiance-mxfp4]] | ACTIVE frank-medium runbook — vllm-radiance serving hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 (ParoQuant MXFP6 W4A8,… |
|
||||
| [[frank-medium-rocmfpx]] | charlie12345/ROCmFPX llama.cpp serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF as frank-medium on Dankserver… |
|
||||
| [[frank-npu]] | The retired FastFlowLM endpoint on Dankclaw's XDNA2 NPU — what it served, why it died on 2026-09-11 with the… |
|
||||
| [[frank-redo-2026-08-17]] | This page records the full redo of the inference fleet on 2026-08-17. |
|
||||
| [[ikllama-ornith-dankserver]] | ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-zealot on the RTX 3090. Q8_0 model, KV K default f16 / V… |
|
||||
| [[kingjones-q8agent-rocmfpx]] | **Status (historical, 2026-09-13):** was to deploy on Dankclaw. |
|
||||
| [[qwen3.8-spec-matrix]] | This page records the speculative-decoding benchmark history and the |
|
||||
|
||||
CLI[LLM Clients / API] -->|HTTP :7770| B2
|
||||
CLI -->|HTTP :7770| E
|
||||
CLI -->|HTTP :7771| G
|
||||
```
|
||||
## Pipeline Parallelism (series)
|
||||
|
||||
## Servers
|
||||
| Page | Owns |
|
||||
|---|---|
|
||||
| [[Ethernet Bandwidth Math]] | Bytes-per-token math for pipeline vs tensor parallelism, feasibility tables for 1/2.5/10 GbE, and real multi-node… |
|
||||
| [[Expected Performance]] | Estimated tokens per second for a 25-40B model split across two nodes over 1/2.5/10 GbE, with real benchmark anchors. |
|
||||
| [[llama.cpp RPC]] | llama.cpp distributed inference via ggml-rpc-server and the --rpc flag: architecture, CLI, versions, limitations, and… |
|
||||
| [[Mixed-Vendor Experiments]] | Real-world NVIDIA + AMD Strix Halo multi-node experiments: what works (llama.cpp RPC, Vulkan split), what breaks… |
|
||||
| [[Qwen3.6 27B Spec Methods]] | Qwen3.6-27B Q8 GGUF variants and llama.cpp speculative decoding (draft-mtp, draft-dflash, ngram-mod): flags, RPC… |
|
||||
| [[Qwen3.8 PP Deployment]] | Split Qwen3.8-27B across the Strix Halo iGPU (head) and the RTX |
|
||||
| [[Sample Script Set]] | An educational two-node llama.cpp RPC split of Qwen3.6-27B-Q8_K_P over ethernet. For reading only. Do not run or… |
|
||||
| [[vLLM Pipeline Parallelism]] | vLLM multi-node pipeline parallelism: flags, Ray/mp backends, interconnect requirements, PP bubble, and PP +… |
|
||||
|
||||
| Server | GPU | VRAM / Memory | OS |
|
||||
| --------------------------------------- | ----------------------------- | -------------------- | ------------ |
|
||||
| [[Dankserver System Specs\|Dankserver]] | 2x AMD Radeon AI PRO R9700 | 2x 32 GB VRAM, 246 GB RAM | Ubuntu 26.04 |
|
||||
| [[Dankclaw System Specs\|Dankclaw]] | AMD Radeon 8060S (Strix Halo) | gfx1151 unified memory | Ubuntu 26.04 |
|
||||
## Techniques, hardware, trials
|
||||
|
||||
## Active Deployment
|
||||
|
||||
- [[frank-dragoon-flash-server-v2-native]] — **ACTIVE since 2026-10-17**
|
||||
(root systemd unit + podman, port 7770): the digest-pinned
|
||||
`ghcr.io/peonist-ai/halogen-flash-server` 0.16.2 engine serving the
|
||||
**native v2 conversion** of the abliterated Qwen3.8-Flash-Next
|
||||
(`qwen38-flash-next-v2-abliterated.hgn`, 66,687,678,432 B, Hub LFS
|
||||
sha256-verified at fetch) with the REQUIRED n-gram lookup table (51.2 GB,
|
||||
@peonist 5cc17cea) and the vision tower (897.9 MB, @e053f488)
|
||||
**hardlinked** beside it — the card publishes only the main file, and the
|
||||
engine's own tools certified the stock companions (`niah` 18/18, `ppl`
|
||||
3.968 with mean KL 0.155 against the Swift1.5 logits on the same corpus).
|
||||
`HALOGEN_CTX=200704` over `HALOGEN_KV_POOL_POSITIONS=401408` (**2 × ctx**):
|
||||
**one pool behind two slots**, sized so TWO maximum-length requests coexist —
|
||||
raised from 200,704 (5.5 GiB) on 2026-10-18 for frank-zealot parity on total
|
||||
in-flight context and **deployed the same evening**: the boot ledger reserves
|
||||
`401408 positions (about 11.0 GiB)` with `79.6 GiB in all`
|
||||
(`scripts/frank-dragoon.env:70-75`). It bought CONCURRENCY, not a longer
|
||||
window: the per-request ceiling stays 200,704, because 262,144 is the
|
||||
native wall and the engine refuses more without `HALOGEN_ROPE_YARN`.
|
||||
MTP head always on, sampler 1.0/0.95/20, coupled cap/timeout
|
||||
65536/7200, prompt cache ON, vision ON. Measured against the Swift1.5
|
||||
boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18,
|
||||
PPL +2.3 %, thinking tokens ×1.9 (the finetune's compression is gone).
|
||||
Rollback: revert the swap commit; the swift15 payload and its fetch script
|
||||
stay on disk
|
||||
- [[frank-dragoon-flash-server-v2-swift15]] — **RETIRED 2026-10-06 → 10-17
|
||||
(rollback target: payload + fetch script on disk)**: the Swift1.5-finetuned
|
||||
v2 `.hgn` (compacted r2, 66,687,678,464 B) at TWO slots at ctx 262,144 over
|
||||
a 524,288-position pool; owns the 400-zone law (the harness chars/4
|
||||
under-count against the engine ceiling), the MTP-always-on law, and the
|
||||
sampler / prompt-cache / vision contract the native era carries verbatim
|
||||
- [[frank-dragoon-halogen27b-dflash2]] — **RETIRED 2026-10-05 → 10-06
|
||||
(rollback target; payload + fetch script + image on disk)**: the
|
||||
digest-pinned `ghcr.io/peonist-ai/halogen` 0.1.4 engine serving
|
||||
`peonist-ai/halogen-qwen3.8-27b` (`qwen3.8-27b-p1w4d-d2.hgn`, 33.4 GiB)
|
||||
— DFlash2 drafter, ONE 200,700 slot, text-only (the engine ignored the
|
||||
vision encoder), censored base. Owns the measured lessons: the prompt
|
||||
cache pin (`HALOGEN_CACHE_MB=16384` — auto was OFF at the 36.9 GB
|
||||
two-entry floor; a 17.3k cold prefill of 27.0 s resumed in 1.7 s warm),
|
||||
the FA tile rejection (`HALOGEN_FA_TQ=256`: prefill −9.5 %, decode
|
||||
±0.3 %), and deep decode at 3.3-3.8 t/s@80k being structural. The
|
||||
foreground-`podman run` ledger capture and the deep-role rerouting
|
||||
(`slow`/`advisor` → frank-archon) carried forward. Superseded by the
|
||||
MODEL swap: the abliterated v2 .hgn loads only in flash-server, so the
|
||||
engine moved with the model
|
||||
- [[frank-zealot-llama-prism-bonsai9-mtp]] — **ACTIVE since 2026-10-18**
|
||||
(root systemd unit, port 7771): the **PrismML fork** build of llama.cpp
|
||||
(vendor **10777**, commit `e3de71522` — the only loader for the payload's
|
||||
custom PQ2_0 tensor type + `prism.hadamard.*` transform, and since this swap
|
||||
also the minimum build that pairs MTP with a quantized V cache) serving
|
||||
prism-ml/Ternary-Bonsai-2-9B-gguf **PQ2_0 with the MTP head IN the file** on
|
||||
the RTX 3090 — **TWO 262,144-token conversations**
|
||||
(`--ctx-size 524288 --parallel 2`, every slot at the window the model was
|
||||
trained at), **unquantized f16 K + f16 V** (32,768 B/token over the 8 full
|
||||
layers = **16,384 MiB** exact; the POOL halved because that rate over the
|
||||
OLD 786,432 pool bills 24,576 MiB against the 24,123 the driver leaves
|
||||
addressable, and the old "26,112 MiB" note was wrong — it doubled the
|
||||
q8_0 figure (17,408 B/token = 13,056 MiB) *including* its 1/32 scale
|
||||
overhead, so f16/q8_0 is **1.882×**, not 2×), the q8_0-era boot settled
|
||||
**17,876 used / 6,249 free**, vision via the publisher's **Q8_0** mmproj
|
||||
(Q8_0 is the publisher's requirement for concurrent image requests, and this
|
||||
runs two slots), **MTP OFF by measurement** — the draft context bills its own
|
||||
f16 KV over the WHOLE pool (4,096 B/position × 524,288 = 2,048 MiB) and at
|
||||
`UBATCH_SIZE=2048` that squeeze halves decode (167.2 → 109.2 t/s);
|
||||
`SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` is the documented 213.5 t/s arm,
|
||||
declined for +17 % prefill and 6.2 GiB of headroom — and exact KV leaves an
|
||||
f16-draft arm no room at all (a quantized draft, `--spec-draft-type-* q8_0`,
|
||||
is the unmeasured door).
|
||||
Measured at the q8_0-era shape: 167.2 t/s single-stream decode, 2,877 t/s
|
||||
prefill at 106,418, 280.2 t/s aggregate across three streams. `--metrics`
|
||||
rides (`METRICS=1`):
|
||||
llama.cpp answers 501 on `/metrics` without it, which is why Prometheus
|
||||
`up{job="frank-zealot"}` read 0 for its whole retained history until
|
||||
2026-10-18. `contextWindow: 262144` is the **widest in the fleet**, so this
|
||||
is the compaction hub. The `--presence-penalty` opt-out law (`${VAR=}`, not
|
||||
`${VAR:=}`) and the `TimeoutStopSec=180` stop-drain law were both earned here.
|
||||
- [[frank-zealot-llama-prism-bonsai2-pq20]] — **RETIRED 2026-10-17 → 10-18**
|
||||
(payload + fork build 10754 tree on disk as rollback): the first PrismML-fork
|
||||
era — prism-ml/Ternary-Bonsai-2-**27B**-gguf **PQ2_0**, TWO 200,704-token
|
||||
conversations (`--ctx-size 401408 --parallel 2`), q8_0 K + q8_0 V
|
||||
(34,816 B/token over the 16 full layers = 13,328 MiB; f16 needs 25,088 MiB and
|
||||
dies), settled 22,664 used / 1,461 free, MTP **impossible** (that export
|
||||
carries no head), 76.6 t/s decode, 691.7 t/s prefill at 166k, 109.4 t/s
|
||||
aggregate at two streams. Owns the 34 KiB/token KV law, the f16 OOM wall, the
|
||||
two-boot VRAM pricing, and the record of the 2026-10-18 bench window
|
||||
(production unit stopped 03:38, hand-booted 9B candidate answering the port)
|
||||
that closed with this swap.
|
||||
- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — **RETIRED 2026-10-07 →
|
||||
10-17** (payload + master CUDA build on disk as rollback): llama.cpp master
|
||||
(CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive
|
||||
**Q6_K** on the RTX 3090 — TWO 200,704-token conversations
|
||||
(`--ctx-size 401408 --parallel 2`), exact f16 K + f16 V, vision via the
|
||||
publisher's BF16 mmproj. MTP is OFF because the file has no MTP tensors at
|
||||
all. Priced from the MiMo Q6_K boots (same qwen35 geometry, same card, same
|
||||
shape). Supersedes the Gemma4 era (below).
|
||||
- [[frank-zealot-llama-gemma4-q4km-200704]] — **RETIRED 2026-10-07** (same
|
||||
day; superseded by qwen35-9b-q6k; payload on disk as rollback): llama.cpp
|
||||
master (CUDA sm_86) serving HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-
|
||||
HauhauCS-Balanced-MTP **Q4_K_M** on the RTX 3090 — ONE 200,704-token
|
||||
conversation, f16 K + q8_0 V, vision via the publisher's mmproj. MTP was
|
||||
OFF (the gemma4-assistant drafter costs ~1.9 GiB: 23,806 used/319 free with
|
||||
it, 21,890/2,235 without).
|
||||
- [[frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688]] — **RETIRED
|
||||
2026-10-07** (superseded by gemma4-q4km; payload on disk as rollback);
|
||||
ACTIVE 2026-10-05 (evening) → 2026-10-07; UNCECERSED RCO-IQ3_XXS with
|
||||
MTP back on since
|
||||
2026-10-06 evening; RENAMED from frank-small and reshaped to 2 ×
|
||||
114,688 / ubatch ladder 512 → 1024 → **2048** late 2026-10-06** (root
|
||||
systemd unit, port
|
||||
7771): llama.cpp master
|
||||
(CUDA sm_86) serving ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-
|
||||
GGUF **RCO-IQ3_XXS-mtp** (27B; the MTP head in the file is DRAFTED —
|
||||
draft-mtp depth 3, ngram skipped on the recurrent-rollback law) on the
|
||||
RTX 3090, TWO 114,688-token conversations at Frank's 112k cap (pool
|
||||
229376 / parallel 2 — cut from 131,072 to fund the ubatch rungs;
|
||||
**1,005 MiB free at boot, 763 settled after a vision probe**) with
|
||||
q8_0 K + q8_0 V (FP8 class, 34 KiB/token vs 64 at f16) and
|
||||
vision through the publisher's own mmproj-BF16.gguf. Prefill 1,209
|
||||
t/s @ 3,760; decode ~52–58 t/s recipe (ubatch-invariant; acceptance
|
||||
noise 34–54 %).
|
||||
RCO-IQ3_XXS tier is a per-tensor allocation mix of stock types; the
|
||||
payload is the UNCECERSED Swift 1.5 (23/100 refusals vs Swift 1.5's
|
||||
98/100). Replaces BOTH same-day payloads: the live MiMo Q6_K shape
|
||||
(below) and the Ornith-1.5-9B Q8_0 swap that was committed
|
||||
(77decb5..380aafa) but never deployed
|
||||
- [[frank-zealot-llama-mimo-q6k-2x200704]] — **RETIRED 2026-10-05 (evening;
|
||||
rollback target; payload on disk)**: llama.cpp master (CUDA sm_86)
|
||||
serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, TWO slots of
|
||||
200,704 with FULLY UNQUANTIZED KV (f16 K + f16 V) and the mmproj for
|
||||
vision — 94.4 tok/s decode (the rung costs a measured -10 % vs Q5_K_M:
|
||||
weight-bandwidth-bound), batch 4096 / ubatch 2048 since 2026-10-05
|
||||
(+4.6 % prefill, measured), 2,279 MiB of the card left free. This was
|
||||
the LIVE payload when Swift cut over. The ubatch A/B table and the
|
||||
probe-size trap (>=256 px images or the model says "video") live inside
|
||||
- [[frank-zealot-llama-mimo-q5km]] — **RETIRED 2026-10-02 → 10-03 (rollback
|
||||
target; payload on disk; owns the measured boot-pair table)**: the same
|
||||
runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower —
|
||||
104.8 tok/s, 3,571 MiB true free
|
||||
- [[frank-archon-radiance-qwen38-iq4r]] — **ACTIVE since 2026-10-15**:
|
||||
the engine author's published container of the OFFICIAL (base-alignment)
|
||||
Qwen3.8-Flash-Next — `qwen3.8-next-flash-fp8-iq4r-moe.rad`, 116,322 MiB,
|
||||
Hub-sha256-verified at fetch: 4-bit codes into a 16-level non-uniform
|
||||
codebook after a Walsh–Hadamard rotation (GPTQ + 3 coordinate-descent
|
||||
sweeps, 10M-token calibration), ten bf16-protected experts per layer,
|
||||
int8 trunk/lm_head, fp8 HC + PLE, MTP head, bf16 vision tower; card KL
|
||||
0.0789 all-tokens / top-1 92.85 % vs bf16. **MODEL_FILE is the only line
|
||||
that moved** — the 2026-10-13 shape (25600/5120 pair, 44928 pool, fp8 KV
|
||||
3 × 200,000, MTP 2, wht6, MNBT 2048, froggeric v22.5) carries verbatim.
|
||||
Rollback: revert the swap commit (w4dn8 `.rad` stays on disk). The
|
||||
fp8-native `.rad` and the raw Swift1.5 BF16 source were DELETED from
|
||||
dankserver 2026-10-15 (Frank's one-time rm, volume for this payload;
|
||||
rebuild via the fetch + build scripts named in the fp8-native runbook)
|
||||
- [[frank-archon-radiance-swift15-w4dn8]] — **RETIRED 2026-10-08 → 10-15
|
||||
(superseded by the publisher iq4r swap; the `.rad` was PURGED from disk
|
||||
2026-10-17 — rollback = revert + byte-exact re-download of
|
||||
`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`)**:
|
||||
Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` on radiance
|
||||
1.0.13 (recipe + 10M-token calibration + KLD gate vs the BF16 container);
|
||||
the FP8-native payload was the second rollback until it too was deleted
|
||||
2026-10-15. **2026-10-13: fp8 KV, and the
|
||||
pair re-cut twice** — `--kv-cache-dtype fp8` first at
|
||||
`--max-num-seqs 4`/6912 MiB pool (843,566 tokens, slab 22208), then **3 slots
|
||||
with the pool cut to the contract (5120 MiB, 608,027 tokens) and the slab
|
||||
raised 22208 → 24000 → 25600**, the last 1,600 MiB claimed out of the card's
|
||||
settling slack: the engine's ceiling is **30,969 MiB**, not the 29,120 my
|
||||
1 GiB reserve invented. fp8 is what funds the slots — the pool must back
|
||||
`seqs × window`, and at bf16 four slots cost 11,679 MiB (slab 17,441, −33 %).
|
||||
Measured on the slab: A-cold 43.5k prefill 22208 → 3,397, 24000 → 3,290,
|
||||
**25600 → 3,584 t/s**; 3-way 197k aggregate 3,051 → **3,589 t/s**. A shipped
|
||||
prediction that the slab was a threshold below 26 GiB died on its own
|
||||
measurement — it is a slope, and 26,071 (4,379 t/s) is the cliff. The bf16
|
||||
arm (2 × 200,000, 462,752 tokens, prefill −29 %, acceptance 0.661 → 0.740)
|
||||
and the 4-slot fp8 shape are the documented rollbacks.
|
||||
**2026-10-13 also: the pinned-ledger preflight now raises `ttm.pages_limit`
|
||||
itself** — the 2026-10-12 power failure
|
||||
reset the knob to its default (125,988 MiB) and the endpoint refused to boot
|
||||
6,919 times in ~20 h; `scripts/ttm-preflight.sh` is the repair.
|
||||
- [[frank-archon-radiance-qwen38-native-fp8]] — **RETIRED 2026-10-08 (the
|
||||
w4dn8 swap replaced it; its `.rad` was DELETED from disk 2026-10-15 —
|
||||
rebuild = `scripts/download-frank-archon.sh` + `scripts/build-frank-archon-rad.sh`,
|
||||
`scripts/frank-archon.env:59-62`)**
|
||||
(root systemd unit, port 7770): the StillDeadcode **radiance** engine
|
||||
(C++/HIP, libr4d gfx1201 kernels, digest-pinned 287 MiB image, engine as
|
||||
the image entrypoint) serving the OFFICIAL `Qwen/Qwen3.8-Flash-Next-FP8`
|
||||
checkpoint at NATIVE precision — converted no-recipe to
|
||||
`qwen4exp-fp8-native.rad` (~161 GiB) after dropping `quantization_config`
|
||||
in a staging dir (the plugin descriptor-selection law: nothing claims
|
||||
`(qwen4exp, "fp8")`, the shipped plugin claims `(qwen4exp, "")`;
|
||||
per-weight encodings are data, not the descriptor) — on BOTH R9700s
|
||||
**TP=2** (`--tp-wire wht6` fused all-reduce), experts in a 40 GiB/rank
|
||||
pinned host pool + VRAM cache (ratio 0.82 = the measured 99 %-hit point),
|
||||
the 47.7 GiB PLE n-gram table PINNED to host RAM (`--ngram-placement
|
||||
ram`), fp8 KV 7.32 KiB/token, MTP spec decode depth 3, 2 × 200,000 first
|
||||
boot, froggeric v22.5 template, vision + tools + `reasoning_content` auto,
|
||||
`/metrics` scraped. Rollback: git tag
|
||||
`frank-archon-freetoken-pr587-20261005` (FreeToken image + orcarouter
|
||||
payload on disk); older rollback `frank-archon-vllm-shapeG-20261003`
|
||||
- [[frank-archon-freetoken-pr587]] — **RETIRED 2026-10-05 (same day;
|
||||
rollback target)**: FreeToken at PR #587 head `77a7201` (branch
|
||||
rocm714_torch214) built on the host into the local image
|
||||
`frank-archon-freetoken:pr587-77a7201` (Dockerfile: digest-pinned
|
||||
rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev-symlink law,
|
||||
curated --no-deps install per docs/install_amd.md), serving the SAME
|
||||
orcarouter FP8 payload on ONE R9700 — experts in pinned host RAM
|
||||
(~120 GiB + 47.7 GiB PLE), GPU LRU at 0.10, KV pool 299,968 tokens
|
||||
(multiple-of-64 law), 2 slots, parsers AUTO (thinking →
|
||||
`reasoning_content`). Measured: boot ~2.3 min, decode 17.1 tok/s
|
||||
single-stream, vision "Red" probe passes, native tool_calls; TP=2 does
|
||||
not boot on ROCm (pynccl JIT) — second card idle. Superseded by the
|
||||
radiance swap for TP=2. Disk state 2026-10-18: the image is still in
|
||||
dankserver's rootless storage (`localhost/frank-primary-freetoken:pr587-77a7201`,
|
||||
29.7 GB — the pre-rename name); the orcarouter FP8 payload is NO LONGER on
|
||||
disk (no `*orcarouter*` path under /home/dank, `find` 2026-10-18)
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **RETIRED 2026-10-05
|
||||
(rollback target; the vLLM trio + env lore live in the tag)**: the
|
||||
tcclaviger RDNA4
|
||||
vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both
|
||||
R9700s, TP=2 with automatic expert offload — `--max-model-len 200704` x 3
|
||||
slots, fp8 KV, pool 3,150,000,000 B/rank (measured 398,658 tokens = 1.99x),
|
||||
MNBT 1024, gather 32x64, no spec decode, prefetch off, plus
|
||||
`--expert-precise-memory` (arena 17.22 GiB / 7,522 slots), GPU_MAX_HW_QUEUES=2
|
||||
and a mounted compile cache. Why `--max-num-seqs`
|
||||
must stay 3 (SEQS 2 cannot compile this model), why the arena's levers are
|
||||
`max_model_len` and the overhead term rather than the KV pool, and the measured envelope
|
||||
- [[frank-archon-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
|
||||
failed GLM experiment; frank-archon runs Qwen FP8 again, see the intro):
|
||||
the tcclaviger RDNA4 vLLM fork
|
||||
(digest-pinned `tcclaviger/vllm@sha256:da4029666…`, plain `podman run`,
|
||||
no shim) serving Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric
|
||||
int4 group 128, ~170 GiB, 34 shards + 2 extras, 113,074 tensors — on BOTH
|
||||
R9700s, TP=2, with the MoE experts offloaded to pinned host RAM (12,096
|
||||
experts/rank, 73.64 GiB) behind a 14.51 GiB / 2,383-slot VRAM LRU arena; the
|
||||
MoE kernel is libr4d's grouped W4A16 GEMM (`R4dW4a16MoEMethod`, reached
|
||||
because the checkpoint's `quant_method: auto-round` is overridden to `inc` —
|
||||
a compressed-tensors W4A16 checkpoint of the same model has NO offload recipe
|
||||
and refuses to serve). 131,072 ctx x 2 seqs, bf16 KV, util 0.98, prefix
|
||||
caching, vision and `glm47` tool/reasoning parsers, MTP OFF and prefetch OFF
|
||||
by measurement. **Measured decode 9.9-10.0 tok/s, prefill 918 tok/s — the
|
||||
35 tok/s target is NOT met, bounded by the fork's expert-gather read path
|
||||
(~8.2 GB/s effective vs a 22.74 GB/s link) at a 0.53-0.66 cache hit rate.**
|
||||
Rollback: commit `2339fae` (the qwen FP8 + MTP4 shape of 2026-09-29 → 09-30)
|
||||
[[frank-archon-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
|
||||
back; the R9V state dirs are on disk (`/home/dank/r9v-state`,
|
||||
`~/.local/state/r9v`) but the payload GGUFs are not — `/home/dank/models`
|
||||
holds only the iq4r `.rad` as of 2026-10-18; git tag
|
||||
frank-archon-r9v-q8-20260929 is the last pre-cutover commit):
|
||||
[[frank-archon-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
|
||||
(same-day swap to the Q8_0 payload; state + payloads no longer on disk
|
||||
2026-10-18, only the R9V state dirs remain): the R9V radiance-vLLM fork serving
|
||||
Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) +
|
||||
blockfp8 MTP4 (depth 4) + F16 vision on BOTH R9700s — TP=2 tiered
|
||||
experts (UVA pinned host copy, 62 hot + 160-slot LRU on rank 0, 428
|
||||
static on rank 1), PLE n-gram on ssd-mmap, CED on, 196608 ctx. Cut over
|
||||
from llama.cpp the same day; rollback target
|
||||
[[frank-archon-llamacpp-qwen38-flash-next-q8]]
|
||||
- [[frank-archon-llamacpp-qwen38-flash-next-q8]] — RETIRED 2026-09-27
|
||||
(rollback target; the llama.cpp b11030 endpoint of 2026-09-26):
|
||||
unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X, bundled runtime)
|
||||
serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 across both R9700s —
|
||||
routed experts and the 51B-param n-gram table computed in host RAM
|
||||
(`-ot exps=CPU,per_layer_token_embd.weight=CPU`), the shared Q8_0 MTP head
|
||||
(depth 2) chained behind ngram-mod, BF16 vision, one full 262,144-token
|
||||
conversation, `reasoning_content`, the placement law, the cutover and
|
||||
rollback
|
||||
- [[frank-archon-lucebox-ds4]] — RETIRED 2026-09-26 (same-day swap,
|
||||
rollback target; active for hours): the Lucebox DS4.1 engine
|
||||
(localhost/lucebox-rocm:gfx1201) serving
|
||||
Lucebox/DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB) with streamed experts —
|
||||
the fleet's only non-llama.cpp runtime detour
|
||||
- [[dankserver-frontier-moe-feasibility]] — VERIFIED 2026-09-28: which of
|
||||
DeepSeek-V4.1-Flash (552B), MiMo-V2.6-Flash-MOPD (309B) and GLM-5.3-Flash
|
||||
(320B) can run on the 2x R9700 box, on what runtime, and why V4.1 does not
|
||||
fit (475 GiB payload, 189 GiB of it Engram; decode is PCIe-expert-stream
|
||||
bound, ~23 tok/s ceiling at today's residency). Includes the Affinity
|
||||
V4-Flash-to-V4.1 port assessment — reusable expert tiering, four
|
||||
architecture gaps, one of them a silent-wrongness trap on the Engram key
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — VERIFIED 2026-09-29:
|
||||
the flag-level decode-tuning campaign on the ad hoc qwen-fp8 container
|
||||
(tcclaviger RDNA4 fork, orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8,
|
||||
TP=2 across both R9700s). **>35 tok/s MET: 37.44-40.57 tok/s with
|
||||
`--expert-prefetch` DELETED** — telemetry measured the prefetcher at
|
||||
36% precision, wasting ~half the gather traffic (the LRU itself runs at
|
||||
82.45% hit). Prefetch-on incumbent 26.8 survived every other challenger
|
||||
(geometry, spec depth, TunableOp-corrupts, batch knobs, concurrency,
|
||||
ngram-fork-bug); the residual gap to ~60 tok/s is the fork's gather
|
||||
read path (6.5-9.5 GB/s effective vs 22.7 at the same slab size)
|
||||
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26 (replaced by
|
||||
[[frank-archon-lucebox-ds4]], then
|
||||
[[frank-archon-llamacpp-qwen38-flash-next-q8]]): vllm-radiance 0.9.3
|
||||
serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark
|
||||
W4A8) with the z-lab DFlash2-FP8 drafter as ONE tensor-parallel process
|
||||
(TP=2) behind :7770, on unquantized bf16 KV (`KV_DTYPE=auto`, cut
|
||||
2026-09-21). The FP8 checkpoint is the on-disk rollback. The three-arm
|
||||
battery that chose TP=2, the auto-sized KV pool, and the verified
|
||||
capability battery (tools, reasoning)
|
||||
- [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium:
|
||||
the streamlining toolchain (hf-manifest.py, hf-fetch.sh,
|
||||
swap-frank-medium.sh), the quantization-aware entrypoint contract, the
|
||||
quadlet edit, the deploy, the capability battery, and the rollback path.
|
||||
The 2026-09-20 MXFP4 → FP8 swap is the worked example
|
||||
- [[frank-medium-llamacpp-dflash2]] — RETIRED 2026-09-16 → 2026-09-20: mainline
|
||||
llama.cpp serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the
|
||||
DFlash2 Q4_K_M drafter as ONE layer-split process (32:33 over two
|
||||
163,840-token slots); superseded by the radiance TP=2 cutover, and the
|
||||
rollback target. Keeps the batch/ubatch optimum curve and the per-layer split
|
||||
arithmetic
|
||||
- [[Dual-R9700-Cooperation]] — whether the two R9700s can serve ONE model: the
|
||||
RCCL/PCIe-atomics wall on the old TUF GAMING B650-E, the 2026-09-20 ProArt
|
||||
B850-Creator x8/x8 swap that cleared it, and the measured three-way verdict
|
||||
(radiance TP=2 95.8 t/s decode vs layer split 45.7 vs llama tensor mode 30.5)
|
||||
that put TP=2 in production
|
||||
- [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era,
|
||||
2026-09-15): charlie12345/ROCmFPX serving
|
||||
kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the
|
||||
runtime it replaced
|
||||
- [[freetoken-frank-small]] — RETIRED 2026-09-15, when the RTX 3090 moved to
|
||||
frank-dragoon: the FreeToken v0.1.2 frank-zealot runbook (batsclamp
|
||||
Huihui-Qwen3.6-35B-A3B-abliterated-FP8), kept as the retirement record. The
|
||||
NAME was reused on 2026-09-16 by a different service on Dankclaw — see
|
||||
[[frank-zealot-qwen38-gsq-rco]]
|
||||
- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep
|
||||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||||
- [[deployment]] — deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[frank-dragoon-rocmfpx-q6agent-uncensored-mtp]] — **RETIRED 2026-10-03 →
|
||||
10-05 (rollback target; payload, fork build and build script on disk)**:
|
||||
the pinned ROCmFPX fork serving the uncensored protected-heads
|
||||
Q6_0_ROCMFPX_AGENT rung with the repo's MTP-Q4_0 head drafting 4 tokens,
|
||||
one 262,144 window, f16 KV; measured 27.6-28.1 t/s prose decode at 86.8 %
|
||||
acceptance. Replaced by the halogen DFlash2 endpoint; the ngram-mod-onto-MTP
|
||||
rejection (-15 %, recurrent-state rollback) stays a live lesson for this
|
||||
architecture.
|
||||
- [[frank-dragoon-rocmfpx-q8agent-mtp]] — **RETIRED 2026-10-02 → 10-03 (the
|
||||
rollback target; payload still on disk)**: the same pinned ROCmFPX fork
|
||||
serving the ALIGNED `Q8_0_ROCMFPX_AGENT` (ftype 115, 17.0 t/s, MTP
|
||||
acceptance 0.462) — the MTP-vs-none +123 % A/B, the trap catalogue
|
||||
- [[frank-dragoon-gufo-qwen38-27b]] — **RETIRED 2026-10-02 (history; the
|
||||
payload dirs were removed from disk 2026-10-03, so this rollback is a
|
||||
re-download)** (root systemd unit, port 7770): gufo
|
||||
(gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from
|
||||
source on Dankclaw, serving unsloth/Qwen3.8-27B-GGUF **UD-Q4_K_XL**
|
||||
with the z-lab DFlash2 Q4_K_M drafter — ONE session at the full native
|
||||
262,144 window, unquantized f16 KV. Rolled back by Frank after one day
|
||||
of [[frank-dragoon-vllm-mimo-v26-bf16]]. The two from-source build
|
||||
workarounds, the client-contract delta, the measured prefill/decode
|
||||
trade against llama.cpp
|
||||
- [[frank-dragoon-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (one day;
|
||||
rollback by Frank after the autoresearch loop): the digest-pinned
|
||||
kyuz0/vllm-therock-gfx1151 image serving
|
||||
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16, 4 x 262144 ctx,
|
||||
CROSS-MODEL MTP from Qwen/Qwen3.5-9B — kept for the measured record:
|
||||
battery decode 6.19 t/s (MTP-2) vs 9.01 t/s (no draft — spec NET-
|
||||
NEGATIVE on real chat+reasoning traffic), the roof math, the greedy-text
|
||||
numerics trap, and the death of every ≥6-bit vLLM quant path on gfx1151
|
||||
(quark int8 rejected, FP8 kernels CDNA-only, torchao unwired)
|
||||
- [[frank-dragoon-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
|
||||
[[frank-dragoon-gufo-qwen38-27b]]; rollback target): stock llama.cpp
|
||||
(unsloth b11030, ROCm gfx1151) serving
|
||||
unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2
|
||||
Q4_K_M drafter (depth 4) on the gfx1151 iGPU — one full 262,144-token
|
||||
conversation, q8_0 KV (the build lacks fp8), text-only, Qwen-Sharp
|
||||
v22.4.0 template, the 114 GiB GTT memory model, client contract,
|
||||
troubleshooting
|
||||
- [[frank-dragoon-kairic-edge-qwen38-27b]] — RETIRED 2026-09-21 (same-day
|
||||
swap, replaced by [[frank-dragoon-unsloth-qwen38-27b-q8xl]]; rollback
|
||||
target): the Kairic Edge runtime (ciru-ai/ROCmFPX v1.2, built from
|
||||
source) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (IU4 W4A4, 27B dense)
|
||||
on the gfx1151 iGPU — one full 262,144-token conversation, MTP depth 4 +
|
||||
n-gram drafting, the three PromptForge sidecars, compatibility mode, the
|
||||
114 GiB GTT memory model, client contract, troubleshooting
|
||||
- [[frank-dragoon-unsloth-qwen38-flash-next]] — RETIRED 2026-09-21 (same-day
|
||||
swap, replaced by [[frank-dragoon-kairic-edge-qwen38-27b]]; rollback target):
|
||||
the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving
|
||||
unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (125B MoE) on the gfx1151 iGPU —
|
||||
one 200,000-token conversation, MTP depth 2 via the explicit -md shared-Q8_0
|
||||
head, the 114 GiB GTT memory model, client contract, troubleshooting
|
||||
- [[frank-dragoon-ciru-orca]] — RETIRED 2026-09-21 (replaced by
|
||||
[[frank-dragoon-unsloth-qwen38-flash-next]]; rollback target): the CIRU
|
||||
v4.4.1 runtime (private host build, QSA hotfix) serving
|
||||
jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU — one
|
||||
262,144-token conversation, MTP depth 4, 4096 MiB PLE cache, runtime-root
|
||||
gate, client contract, troubleshooting (2026-09-19 → 09-21)
|
||||
- [[halogen-flash-server]] — RETIRED 2026-09-19 (replaced by
|
||||
[[frank-dragoon-ciru-orca]]; rollback target): peonist-ai halogen-flash-server
|
||||
0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b
|
||||
.hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
|
||||
400,384-position pool, deployed flags, memory model, client contract,
|
||||
troubleshooting (2026-09-16 → 09-19)
|
||||
- [[frank-zealot-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02** (the same
|
||||
rung returned ACTIVE 2026-10-03 at the 2 x 200,704 f16-KV shape —
|
||||
[[frank-zealot-llama-mimo-q6k-2x200704]]): llama.cpp (master, CUDA sm_86
|
||||
build) serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO
|
||||
native 262,144-token sessions with ASYMMETRIC KV (K f16 unquantized + V
|
||||
q8_0, which forces flash-attn), ~20.9/24.6 GiB measured, text-only (the
|
||||
GGUF has no vision), reasoning_content, no spec decode; payload still at
|
||||
`/home/dank/models/MiMo-Q6K/`
|
||||
- [[frank-zealot-sglang-mimo-v26]] — RETIRED 2026-10-01 (hours; replaced
|
||||
by [[frank-zealot-llama-mimo-q6k]] for the asymmetric KV SGLang cannot
|
||||
express; venv still on disk, INT8 payload staged to
|
||||
`~dank/.AGENTTRASH/20261002-retired-llm-payloads/`): SGLang in a
|
||||
self-contained python3.12 venv serving leteschlead MiMo-V2.6 INT8
|
||||
W8A16 AutoRound, 2 x 262144 ctx, fp8 KV, in-checkpoint vision,
|
||||
83.9 tok/s decode
|
||||
- [[frank-zealot-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by
|
||||
[[frank-zealot-sglang-mimo-v26]] → the llama.cpp MiMo shape; payload staged
|
||||
to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/`):
|
||||
mainline llama.cpp serving
|
||||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
|
||||
3090, with the separate MTP assistant sidecar chained behind ngram-mod,
|
||||
q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB.
|
||||
- [[frank-zealot-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
|
||||
rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
|
||||
Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-
|
||||
token pool, 2 concurrent full-context sessions, native text+image+audio.
|
||||
- [[frank-zealot-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the
|
||||
prior frank-zealot runbook (2026-09-16 → 09-17), ISTA-DASLab
|
||||
Qwen3.8-27B-GSQ-RCO IQ3_S with the model's own in-file MTP head and
|
||||
f16 KV at 131k
|
||||
- [[x4-pair-experiment]] — MEASURED 2026-09-16: what mainline llama.cpp can and
|
||||
cannot do with the 3090 + Strix Halo pair, over their only link (the 3090's
|
||||
Gen4 x4). The tensor inventory that decides any split, the single flag (`-ot`)
|
||||
that kills per-tensor placement across devices, the two-device layer splits
|
||||
that work (+7 % decode at 64k, cross-vendor included), the harness-parsing trap
|
||||
that invalidated the first round, and the iGPU-only result that matches
|
||||
halogen's deployed decode rate — at a memory cost the co-tenants cannot pay
|
||||
- [[frank-dragoon-ik-uncensored]] — RETIRED 2026-09-16 (rollback target): the
|
||||
ik_llama.cpp runbook (root systemd unit, 2026-09-15 → 09-16) serving
|
||||
mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090
|
||||
(dense/attention/KV) + LPDDR (experts/PLE) hybrid; its measured envelope and
|
||||
decode root-cause chain are the record of why the fleet went back to halogen
|
||||
- [[dankclaw-iommu-translation-off]] — Dankclaw's IOMMU boot line: **`iommu=pt`**
|
||||
since 2026-10-18, measured on this host at no prefill cost (+2.0 %, 1,616 t/s
|
||||
cold), which restored the XDNA2 NPU; the `amd_iommu=off` era (2026-09-11) it
|
||||
replaced ran on a 13-16 % figure measured upstream only; includes the AMD
|
||||
PerfOpt research, what was deliberately not copied from the reference machine,
|
||||
and the revert path via `scripts/set-dankclaw-bootline.sh`
|
||||
- [[kingjones-strix-rocmfp4]] — PRIOR frank-dragoon runbook and rollback
|
||||
target: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params
|
||||
(56B MoE + 51.2B PLE), single 113.47 GiB GGUF, on the card's pinned
|
||||
kingjones30/ROCmFPX fork (2026-08-30 → 2026-09-11)
|
||||
- [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
|
||||
- [[frank-medium-batch-ubatch]] — batch/ubatch sweep on the 2x R9700 layer split
|
||||
(2026-09-16): the ubatch optimum at 512, why the logical batch is free, and
|
||||
the split rebalance that followed it
|
||||
- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
|
||||
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model
|
||||
on Dankclaw (retired)
|
||||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on
|
||||
Dankserver (frank-zealot, retired 2026-09-02)
|
||||
- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for
|
||||
frank-zealot (2026-08-26): q8_0 + Hadamard KV cache,
|
||||
split-mode none (single-GPU), mandatory FA_ALL_QUANTS build flags
|
||||
- [[kingjones-strix-mtp]] — the former Q6 AGENT runbook (restored 2026-09-01
|
||||
after Q8_0 failure, then superseded by the STRIX deployment on 2026-08-27)
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime
|
||||
on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
|
||||
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
|
||||
(deployed and reverted 2026-08-22)
|
||||
- [[Dankserver System Specs]] — the 2x R9700 box (frank-archon)
|
||||
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-dragoon on
|
||||
the flash-server v2 runtime since 2026-10-06, frank-zealot on the RTX 3090)
|
||||
- [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint
|
||||
on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as
|
||||
the retirement record — the IOMMU/SVA err=-19 root cause and where
|
||||
every dependent route moved
|
||||
|
||||
## CIRU Reference (Dankclaw)
|
||||
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 model, ROCm 7.15 toolchain,
|
||||
and build history (on Dankclaw; the gfx1151 slot now serves frank-dragoon's
|
||||
flash-server v2-native payload — see [[frank-dragoon-flash-server-v2-native]])
|
||||
- [[kairic-edge]] — Kairic Edge IU4 runtime: build runbook, binary
|
||||
identity, and experiment summary (retired 2026-08-22)
|
||||
|
||||
## DFlash2 Reference (archived)
|
||||
|
||||
- [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment
|
||||
(2026-08-21)
|
||||
|
||||
## vLLM Reference (historical)
|
||||
|
||||
- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
|
||||
- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
|
||||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware
|
||||
notes and sizing (this architecture family is what the retired frank-zealot
|
||||
served in the Huihui abliterated FP8 variant — see
|
||||
[[freetoken-frank-small]]; the Ornith-1.5-35B-A3B CRACK checkpoint
|
||||
that preceded it is archived at [[dealign-ornith-crack]])
|
||||
|
||||
## Memory Management
|
||||
|
||||
- [[RAM CPU Offloading in VLLM]] — KV cache swap between GPU VRAM and system RAM
|
||||
- [[Hybrid CPU GPU Processing Dankserver]] — NVIDIA GPU compute with system RAM offload
|
||||
- [[Shared Memory Architecture Dankclaw]] — AMD unified memory and ROCm configuration
|
||||
|
||||
## Retired Topology (2026-08-17 redo)
|
||||
|
||||
- [[Qwen3.8 PP Deployment]] — the retired pipeline-parallel head + worker
|
||||
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history (superseded by FastMTP)
|
||||
- [[FastFlowLM NPU Inference]] — the retired FastFlowLM NPU endpoint
|
||||
- [[frank-dragoon-benchmark]] — backend/spec benchmark behind the old head
|
||||
- [[dankserver-vram-growth]] — VRAM/RAM growth behind the retired cuBLAS crash
|
||||
- [[DFlash DSpark Speculative Decoding]] — retired DFlash/DSpark drafting
|
||||
- [[Quantization by Hardware Architecture]] — which quant format fits which GPU/CPU
|
||||
|
||||
## Pipeline Parallelism (archive)
|
||||
|
||||
- [[Pipeline Parallelism/index]] — index of multi-node inference research. The PP
|
||||
pair is retired. These pages are research history.
|
||||
|
||||
## Runtime Research (llama.cpp family)
|
||||
|
||||
- [[Llama Cpp Runtimes]] — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: positioning, quantization, server fit
|
||||
- [[Llama Cpp Code Diff]] — code-level quantization diff of the three runtimes + server recommendations
|
||||
- [[SGLang]] — the structured-generation serving engine; when it fits Dankserver (2x R9700) vs Dankclaw (Strix Halo)
|
||||
|
||||
## Cross-Cutting References (skill distillation, 2026-10-05)
|
||||
|
||||
One polished reference page per skill-cluster, distilled from the managed
|
||||
skill library into the vault (see [log](../log.md) for the wave plan):
|
||||
|
||||
- [[Quantization Production]] — how we self-make quants per card
|
||||
(imatrix/K_P ladders, radiance recipes, REAP/Heretic/abliteration
|
||||
prior art), and what each runtime decodes natively
|
||||
- [[GGUF Compatibility]] — what a GGUF's tensor histogram and header demand
|
||||
of a loader; how the runtime is chosen by the file
|
||||
- [[Llama Cpp Serving Sizing]] — slot/pool/KV sizing laws, boot-shape
|
||||
laddering, llama-bench discipline
|
||||
- [[Llama Cpp Multi-Device Splits]] — layer vs tensor splits, `-ts`/`-ot`
|
||||
laws, drafter placement
|
||||
- [[PCIe Link and Atomics]] — the link walls, atomics routing, and the
|
||||
heterogeneous-pair verdicts
|
||||
- [[RDNA4 Container Serving]] — bring-up recipes for RDNA4 containers
|
||||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan law, every arena
|
||||
lever, and why free VRAM is not arena
|
||||
- [[vLLM Boot Forensics and Config Interrogation]] — reading a boot that
|
||||
died, and interrogating a running fork's config
|
||||
- [[vLLM Porting and Compile Triage]] — porting checkpoints across vLLM
|
||||
versions/forks; torch.compile/dynamo breakage triage
|
||||
- [[LLM Bench Windows and Measurement]] — window discipline, honest A/B,
|
||||
the window script laws
|
||||
- [[LLM Performance Diagnosis]] — slow prefill/decode triage on live
|
||||
endpoints, IBS attribution, prefix-cache routing
|
||||
- [[Model Swap and Rollback]] — the swap procedure, payload preflight,
|
||||
research verification, second-endpoint bring-up
|
||||
- [[Tcclaviger Fork Ops]] — the fork's sweep harness, measured knob
|
||||
verdicts, large-MoE recipe, compile-cache persistence
|
||||
- [[Heterogeneous Split Triage]] — triage for unequal-accelerator split
|
||||
proposals (the no-go verdict format)
|
||||
- [[FreeToken Builds]] — the CUDA and ROCm build paths with their
|
||||
dependency traps
|
||||
|
||||
These sit alongside the operational pages under
|
||||
`../Operations/index.md` and `../Agent/index.md`.
|
||||
|
||||
## Related
|
||||
|
||||
- [Project knowledge base](../index.md) — the Podman app stacks on the same
|
||||
two hosts. OpenWebUI (Dankclaw) consumes the on-host `frank-dragoon`
|
||||
endpoint at `:7770`. The Gitea instance on Dankclaw hosts this repository.
|
||||
- [Deployment Flow](../Architecture/Deployment%20Flow.md) — the service-scoped
|
||||
git-based deploy with inference isolation
|
||||
| Page | Owns |
|
||||
|---|---|
|
||||
| [[Dankclaw Improvements]] | This page collects concrete, sourced recommendations to improve Dankclaw |
|
||||
| [[Dankclaw System Specs]] | Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. |
|
||||
| [[dankclaw-iommu-translation-off]] | Dankclaw boots iommu=pt since 2026-10-18 — identity DMA with the IOMMU armed, which restored the XDNA2 NPU at a… |
|
||||
| [[Dankserver System Specs]] | Dankserver hardware and OS state since the 2026-09-20 board swap (ASUS ProArt B850-Creator WiFi NEO, 2x AMD Radeon AI… |
|
||||
| [[dankserver-frontier-moe-feasibility]] | VERIFIED 2026-09-28 by measurement and source reading. Compares DeepSeek-V4.1-Flash, MiMo-V2.6-Flash-MOPD and… |
|
||||
| [[dankserver-vram-growth]] | Investigation of the recurring cuBLAS crash on frank-zealot. What grows, what does not, and what to do next. |
|
||||
| [[dealign-ornith-crack]] | dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with… |
|
||||
| [[deployment]] | Scoped deployment and update process for the three inference services (frank-archon, frank-dragoon, frank-zealot). |
|
||||
| [[DFlash DSpark Speculative Decoding]] | DFlash and DSpark are the two newest speculative decoding methods. |
|
||||
| [[Dual-R9700-Cooperation]] | Why frank-medium's two Radeon AI PRO R9700s could not run tensor-parallel on the old B650-E board (PCIe atomic… |
|
||||
| [[FastFlowLM NPU Inference]] | FastFlowLM (FLM), the AMD NPU-first LLM runtime for Ryzen AI XDNA2 chips. Fit analysis for the 50 TOPS NPU on… |
|
||||
| [[FreeToken Builds]] | Building FreeToken (the official franksdank fork of FreeTok Tokenworks' freetoken vLLM-MTP runtime) on Dankclaw — the… |
|
||||
| [[freetoken-benchmark-sweep]] | KV pool vs decode speed sweep across bs1/bs3/bs4/bs6 on Dankserver with Qwen3.6-35B-A3B-FP8 via FreeToken v0.1.2. |
|
||||
| [[GGUF Compatibility]] | How to answer "will engine X load this GGUF?" from the file's header alone, why a custom tensor type forces the… |
|
||||
| [[Heterogeneous Split Triage]] | Triage before building when someone asks to split a model across unequal accelerators on one host (24 GB GDDR6X +… |
|
||||
| [[Hybrid CPU GPU Processing Dankserver]] | Dankserver uses NVIDIA GPU with system RAM for hybrid processing. |
|
||||
| [[ling-3.0-flash-trial]] | `bailingmoe3-support` (commit `db4480bc8`, build b10411) at `/home/dank/llama.cpp-ling`. |
|
||||
| [[ling-3.0-flash-vllm-plan]] | This page was the plan for a vLLM-ROCm deployment of `jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native` (INT4 W4A16, ~62… |
|
||||
| [[Llama Cpp Multi-Device Splits]] | What llama.cpp can and cannot express across devices, the no-collective layer-split path, split balancing by… |
|
||||
| [[Llama Cpp Serving Sizing]] | Slot-pool sizing against a headroom band, per-device VRAM accounting, prompt-cache/state sizing and the QSA… |
|
||||
| [[LLM Bench Windows and Measurement]] | Stopping-and-restoring benchmark windows on the fleet LLM hosts (llama-bench sweeps, HTTP param benches, multi-arm… |
|
||||
| [[LLM Performance Diagnosis]] | Probing why a served LLM is slow — the flat-depth probe, the expert-offload duty-cycle arithmetic, kernel-time… |
|
||||
| [[Model Swap and Rollback]] | The full model-swap discipline for the LLM fleet — fit preflight, fetch and payload verification, runtime build/flag… |
|
||||
| [[PCIe Link and Atomics]] | Link probing (idle downshift, path ceilings, CUDA/HIP bandwidth probes), the streaming-vs-compute ceiling arithmetic,… |
|
||||
| [[Quantization by Hardware Architecture]] | This page maps LLM quantization formats to the GPU and CPU architectures that support them. |
|
||||
| [[Quantization Production]] | — |
|
||||
| [[qwen3.8-strix-optimizations]] | This page answers one question: **how do we make Qwen3.8-27B serve faster on |
|
||||
| [[RAM CPU Offloading in VLLM]] | vLLM uses CPU RAM and swap space to extend GPU memory capacity. |
|
||||
| [[RDNA4 Container Serving]] | Bringing a model larger than VRAM onto Dankserver's 2x R9700 (gfx1201) with the tcclaviger vLLM stack — quant-class… |
|
||||
| [[RDNA4 MoE Expert Offload VRAM Budget]] | The expert-offload plan arithmetic on the tcclaviger fork — which terms a deploy can move, the util law, the… |
|
||||
| [[Running Qwen3.6 27B FP8 on Dankclaw]] | Qwen3.6-27B is a dense vision-language model from the Qwen3.6 series. |
|
||||
| [[Running Qwen3.6 35B A3B FP8 on Dankserver]] | Qwen3.6-35B-A3B is a Mixture-of-Experts vision-language model from the Qwen3.6 series. |
|
||||
| [[Shared Memory Architecture Dankclaw]] | Dankclaw uses AMD Strix Halo APU with unified memory architecture. |
|
||||
| [[swift15-campaign-worklog]] | — |
|
||||
| [[Tcclaviger Fork Ops]] | Operating the tcclaviger RDNA4 vLLM fork on dankserver (shape G era, rollback target since the 2026-10-05 FreeToken… |
|
||||
| [[ubatch-benchmark-large]] | Standardized ubatch sweep for the active kingjones Qwen3.8 ROCmFPX deployment. |
|
||||
| [[vLLM Boot Forensics and Config Interrogation]] | Capturing a vLLM container boot failure's real root cause from journald (the container log is already gone),… |
|
||||
| [[VLLM Index]] | This index contains all documentation for vLLM deployment on Dankserver and Dankclaw. |
|
||||
| [[VLLM Overview]] | vLLM is a fast and easy library for large language model inference and serving. |
|
||||
| [[vLLM Porting and Compile Triage]] | Forward-porting a model architecture to a new vLLM major (additive-only discipline, gap enumeration, shims that… |
|
||||
| [[vllm_research_findings]] | vLLM v0.26.0 (latest stable, released July 25, 2026) supports DFlash speculative decoding with the `--spec-method… |
|
||||
| [[x4-pair-experiment]] | What mainline llama.cpp can and cannot do with a 3090 + Strix Halo combination, measured 2026-09-16 on Dankclaw — the… |
|
||||
|
||||
Reference in New Issue
Block a user