Frank
2b127163c8
frank-dragoon's total in-flight context was HALF frank-zealot's: zealot runs one 401,408-position pool split into two 200,704 conversations, while dragoon ran ONE 200,704-position pool behind its two slots, so a single maximum-length request filled it and two long conversations had to stay under ~100k each. HALOGEN_KV_POOL_POSITIONS 200704 -> 401408 = slots x ctx, the top of the seam's legal band (ctx <= pool <= slots * ctx). The PER-REQUEST ceiling does not move: HALOGEN_CTX stays 200704, because the checkpoint's native window is 262,144 and the engine's own 400 text refuses to exceed it without HALOGEN_ROPE_YARN. What this buys is concurrency, not a longer conversation, which is why the harness contextWindow (155648 = ceiling - 45056) is untouched. Cost: ~10.8 GiB of KV where 200,704 cost 5.5 (262,144 cost 7.2; the 2026-10-06 shape's 524,288 measured 14.6), taken from the page cache the 51.2 GB n-gram table reads through — the warm prompt-cache ratio is the number to re-measure after the boot. Engine-side total goes 74.1 -> ~79.4 GiB of 121.2. Also carries scripts/download-frank-zealot-bonsai2-ptq1.sh: the PTQ1_0 candidate payload fetcher (5,946,648,928 B, LFS sha256-verified, mmproj hardlinked), with the publisher's own throughput table in its header — the packing is smaller but the card measures it SLOWER on Ampere, so the swap is decided on our card, not on the card's A100 row.
600 lines
37 KiB
Markdown
600 lines
37 KiB
Markdown
# LLM Fleet — Knowledge Base
|
||
|
||
**Dankserver runs the radiance engine (StillDeadcode) on frank-archon**
|
||
(digest-pinned `stilldeadcode/radiance` image, engine as entrypoint,
|
||
driving `podman run` directly, root systemd unit, :7770):
|
||
the **publisher's** 4-bit-codebook container of the official
|
||
`Qwen/Qwen3.8-Flash-Next` (`qwen3.8-next-flash-fp8-iq4r-moe.rad`, 116,322 MiB,
|
||
sha256-verified at fetch — w4nl codebook experts + int8 trunk, PLE n-gram,
|
||
MTP head and vision tower in-container), on BOTH R9700s
|
||
TP=2 (`--tp-wire wht6`), experts in a 44 GiB/rank pinned host pool + VRAM cache
|
||
(explicit `--vram-weights-mib 25600 --vram-kv-mib 5120` = 30,720 of the
|
||
30,969 MiB the engine's own budget table makes claimable), the 47.7 GiB PLE
|
||
table pinned to host RAM, **fp8 KV at 3 × 200,000**, MTP spec decode depth 2,
|
||
froggeric v22.5 template. Rollback: the in-house Swift1.5 w4dn8 `.rad` (on
|
||
disk); tags frank-archon-freetoken-pr587-20261005 and
|
||
frank-archon-vllm-shapeG-20261003),
|
||
and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2
|
||
(frank-dragoon, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned
|
||
digest serving the NATIVE v2 conversion of the abliterated
|
||
Qwen3.8-Flash-Next, a ~62 GiB v2 .hgn that loads in no other engine,
|
||
MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots over ONE
|
||
401,408-position KV pool at ctx 200704, vision ON)** and
|
||
**llama.cpp on the PrismML fork (frank-zealot, port 7771: prism-ml
|
||
Ternary-Bonsai-2-27B at **PQ2_0** — 27B qwen35 hybrid (16 full-attention + 48
|
||
Gated DeltaNet layers), TWO 200,704-token conversations, MTP disabled,
|
||
q8_0 K + q8_0 V (f16 KV does not fit the RTX 3090 at this window), vision via
|
||
the publisher's own Q8_0 Qwen3-VL mmproj, on the RTX 3090 — and the custom
|
||
tensor type 142 + `prism.hadamard.*` metadata mean no other loader can open the
|
||
file)**.
|
||
|
||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||
> `serverAppConfigs-docs/` root to `serverAppConfigs-docs/LLM/`. The fleet scripts and units now live in the
|
||
> repo's `scripts/` directory.
|
||
|
||
## Deployment Topology
|
||
|
||
```mermaid
|
||
graph LR
|
||
subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
|
||
A[2x Radeon AI PRO R9700
|
||
gfx1201, 32 GiB each]
|
||
B2[radiance StillDeadcode, digest-pinned image
|
||
podman run, frank-archon, port 7770]
|
||
C[Publisher qwen3.8-next-flash-fp8-iq4r-moe .rad
|
||
2x R9700 TP=2 wht6, experts 44 GiB/rank pinned pool + VRAM,
|
||
PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 2,
|
||
3 x 200000, froggeric v22.5]
|
||
end
|
||
|
||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||
E2[RTX 3090 (PCIe x4 dock) +
|
||
gfx1151 iGPU 114 GiB GTT]
|
||
E[flash-server container gfx1151, digest-pinned image
|
||
frank-dragoon, port 7770]
|
||
F[flash-server v2 native abliterated .hgn + ngram table + vision tower
|
||
ONE 401408-position KV pool at ctx 200704, 2 slots, MTP always-on, image input]
|
||
G[llama.cpp PrismML fork CUDA build
|
||
frank-zealot, port 7771]
|
||
H[Ternary-Bonsai-2-27B PQ2_0 GGUF + Q8_0 mmproj
|
||
2 x 200704 ctx, q8_0 K + q8_0 V, MTP off, mmproj vision]
|
||
E2 --> E --> F
|
||
E2 --> G --> H
|
||
end
|
||
|
||
CLI[LLM Clients / API] -->|HTTP :7770| B2
|
||
CLI -->|HTTP :7770| E
|
||
CLI -->|HTTP :7771| G
|
||
```
|
||
|
||
## Servers
|
||
|
||
| Server | GPU | VRAM / Memory | OS |
|
||
| --------------------------------------- | ----------------------------- | -------------------- | ------------ |
|
||
| [[Dankserver System Specs\|Dankserver]] | 2x AMD Radeon AI PRO R9700 | 2x 32 GB VRAM, 246 GB RAM | Ubuntu 26.04 |
|
||
| [[Dankclaw System Specs\|Dankclaw]] | AMD Radeon 8060S (Strix Halo) | gfx1151 unified memory | Ubuntu 26.04 |
|
||
|
||
## Active Deployment
|
||
|
||
- [[frank-dragoon-flash-server-v2-native]] — **ACTIVE since 2026-10-17**
|
||
(root systemd unit + podman, port 7770): the digest-pinned
|
||
`ghcr.io/peonist-ai/halogen-flash-server` 0.16.2 engine serving the
|
||
**native v2 conversion** of the abliterated Qwen3.8-Flash-Next
|
||
(`qwen38-flash-next-v2-abliterated.hgn`, 66,687,678,432 B, Hub LFS
|
||
sha256-verified at fetch) with the REQUIRED n-gram lookup table (51.2 GB,
|
||
@peonist 5cc17cea) and the vision tower (897.9 MB, @e053f488)
|
||
**hardlinked** beside it — the card publishes only the main file, and the
|
||
engine's own tools certified the stock companions (`niah` 18/18, `ppl`
|
||
3.968 with mean KL 0.155 against the Swift1.5 logits on the same corpus).
|
||
`HALOGEN_CTX=200704` over `HALOGEN_KV_POOL_POSITIONS=401408` (**2 × ctx**):
|
||
**one pool behind two slots**, sized so TWO maximum-length requests coexist —
|
||
raised from 200,704 (5.5 GiB) on 2026-10-18 for frank-zealot parity on total
|
||
in-flight context, ~10.8 GiB reserved against the card's 262,144/262,144 =
|
||
7.2 GiB. The RAM comes out of the n-gram table's page cache, not out of a
|
||
longer window: the per-request ceiling stays 200,704, because 262,144 is the
|
||
native wall and the engine refuses more without `HALOGEN_ROPE_YARN`.
|
||
MTP head always on, sampler 1.0/0.95/20, coupled cap/timeout
|
||
65536/7200, prompt cache ON, vision ON. Measured against the Swift1.5
|
||
boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18,
|
||
PPL +2.3 %, thinking tokens ×1.9 (the finetune's compression is gone).
|
||
Rollback: revert the swap commit; the swift15 payload and its fetch script
|
||
stay on disk
|
||
- [[frank-dragoon-flash-server-v2-swift15]] — **RETIRED 2026-10-06 → 10-17
|
||
(rollback target: payload + fetch script on disk)**: the Swift1.5-finetuned
|
||
v2 `.hgn` (compacted r2, 66,687,678,464 B) at TWO slots at ctx 262,144 over
|
||
a 524,288-position pool; owns the 400-zone law (the harness chars/4
|
||
under-count against the engine ceiling), the MTP-always-on law, and the
|
||
sampler / prompt-cache / vision contract the native era carries verbatim
|
||
- [[frank-dragoon-halogen27b-dflash2]] — **RETIRED 2026-10-05 → 10-06
|
||
(rollback target; payload + fetch script + image on disk)**: the
|
||
digest-pinned `ghcr.io/peonist-ai/halogen` 0.1.4 engine serving
|
||
`peonist-ai/halogen-qwen3.8-27b` (`qwen3.8-27b-p1w4d-d2.hgn`, 33.4 GiB)
|
||
— DFlash2 drafter, ONE 200,700 slot, text-only (the engine ignored the
|
||
vision encoder), censored base. Owns the measured lessons: the prompt
|
||
cache pin (`HALOGEN_CACHE_MB=16384` — auto was OFF at the 36.9 GB
|
||
two-entry floor; a 17.3k cold prefill of 27.0 s resumed in 1.7 s warm),
|
||
the FA tile rejection (`HALOGEN_FA_TQ=256`: prefill −9.5 %, decode
|
||
±0.3 %), and deep decode at 3.3-3.8 t/s@80k being structural. The
|
||
foreground-`podman run` ledger capture and the deep-role rerouting
|
||
(`slow`/`advisor` → frank-archon) carried forward. Superseded by the
|
||
MODEL swap: the abliterated v2 .hgn loads only in flash-server, so the
|
||
engine moved with the model
|
||
- [[frank-zealot-llama-prism-bonsai2-pq20]] — **ACTIVE since 2026-10-17**
|
||
(root systemd unit, port 7771): the **PrismML fork** build of llama.cpp
|
||
(vendor 10754, commit `2459f68b5` — the only loader for the payload's custom
|
||
tensor type 142 + `prism.hadamard.*` transform) serving
|
||
prism-ml/Ternary-Bonsai-2-27B-gguf **PQ2_0** (2.13 bpw) on the RTX 3090 —
|
||
TWO 200,704-token conversations (`--ctx-size 401408 --parallel 2`), **q8_0 K
|
||
+ q8_0 V** (34 KiB/token over the 16 full layers = 13,328 MiB; the f16 pair
|
||
needs 25,088 MiB and dies), settled 22,664 used / 1,461 free, vision via the
|
||
publisher's Q8_0 mmproj (+873 MiB, charged AFTER the engine's fit check), MTP
|
||
OFF (no MTP tensors). 76.6 t/s single-stream decode, 691.7 t/s prefill at
|
||
166k, 109.4 t/s aggregate at two streams. Supersedes the Qwen3.5-9B era below.
|
||
- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — **RETIRED 2026-10-07 →
|
||
10-17** (payload + master CUDA build on disk as rollback): llama.cpp master
|
||
(CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive
|
||
**Q6_K** on the RTX 3090 — TWO 200,704-token conversations
|
||
(`--ctx-size 401408 --parallel 2`), exact f16 K + f16 V, vision via the
|
||
publisher's BF16 mmproj. MTP is OFF because the file has no MTP tensors at
|
||
all. Priced from the MiMo Q6_K boots (same qwen35 geometry, same card, same
|
||
shape). Supersedes the Gemma4 era (below).
|
||
- [[frank-zealot-llama-gemma4-q4km-200704]] — **RETIRED 2026-10-07** (same
|
||
day; superseded by qwen35-9b-q6k; payload on disk as rollback): llama.cpp
|
||
master (CUDA sm_86) serving HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-
|
||
HauhauCS-Balanced-MTP **Q4_K_M** on the RTX 3090 — ONE 200,704-token
|
||
conversation, f16 K + q8_0 V, vision via the publisher's mmproj. MTP was
|
||
OFF (the gemma4-assistant drafter costs ~1.9 GiB: 23,806 used/319 free with
|
||
it, 21,890/2,235 without).
|
||
- [[frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688]] — **RETIRED
|
||
2026-10-07** (superseded by gemma4-q4km; payload on disk as rollback);
|
||
ACTIVE 2026-10-05 (evening) → 2026-10-07; UNCECERSED RCO-IQ3_XXS with
|
||
MTP back on since
|
||
2026-10-06 evening; RENAMED from frank-small and reshaped to 2 ×
|
||
114,688 / ubatch ladder 512 → 1024 → **2048** late 2026-10-06** (root
|
||
systemd unit, port
|
||
7771): llama.cpp master
|
||
(CUDA sm_86) serving ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-
|
||
GGUF **RCO-IQ3_XXS-mtp** (27B; the MTP head in the file is DRAFTED —
|
||
draft-mtp depth 3, ngram skipped on the recurrent-rollback law) on the
|
||
RTX 3090, TWO 114,688-token conversations at Frank's 112k cap (pool
|
||
229376 / parallel 2 — cut from 131,072 to fund the ubatch rungs;
|
||
**1,005 MiB free at boot, 763 settled after a vision probe**) with
|
||
q8_0 K + q8_0 V (FP8 class, 34 KiB/token vs 64 at f16) and
|
||
vision through the publisher's own mmproj-BF16.gguf. Prefill 1,209
|
||
t/s @ 3,760; decode ~52–58 t/s recipe (ubatch-invariant; acceptance
|
||
noise 34–54 %).
|
||
RCO-IQ3_XXS tier is a per-tensor allocation mix of stock types; the
|
||
payload is the UNCECERSED Swift 1.5 (23/100 refusals vs Swift 1.5's
|
||
98/100). Replaces BOTH same-day payloads: the live MiMo Q6_K shape
|
||
(below) and the Ornith-1.5-9B Q8_0 swap that was committed
|
||
(77decb5..380aafa) but never deployed
|
||
- [[frank-zealot-llama-mimo-q6k-2x200704]] — **RETIRED 2026-10-05 (evening;
|
||
rollback target; payload on disk)**: llama.cpp master (CUDA sm_86)
|
||
serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, TWO slots of
|
||
200,704 with FULLY UNQUANTIZED KV (f16 K + f16 V) and the mmproj for
|
||
vision — 94.4 tok/s decode (the rung costs a measured -10 % vs Q5_K_M:
|
||
weight-bandwidth-bound), batch 4096 / ubatch 2048 since 2026-10-05
|
||
(+4.6 % prefill, measured), 2,279 MiB of the card left free. This was
|
||
the LIVE payload when Swift cut over. The ubatch A/B table and the
|
||
probe-size trap (>=256 px images or the model says "video") live inside
|
||
- [[frank-zealot-llama-mimo-q5km]] — **RETIRED 2026-10-02 → 10-03 (rollback
|
||
target; payload on disk; owns the measured boot-pair table)**: the same
|
||
runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower —
|
||
104.8 tok/s, 3,571 MiB true free
|
||
- [[frank-archon-radiance-qwen38-iq4r]] — **ACTIVE since 2026-10-15**:
|
||
the engine author's published container of the OFFICIAL (base-alignment)
|
||
Qwen3.8-Flash-Next — `qwen3.8-next-flash-fp8-iq4r-moe.rad`, 116,322 MiB,
|
||
Hub-sha256-verified at fetch: 4-bit codes into a 16-level non-uniform
|
||
codebook after a Walsh–Hadamard rotation (GPTQ + 3 coordinate-descent
|
||
sweeps, 10M-token calibration), ten bf16-protected experts per layer,
|
||
int8 trunk/lm_head, fp8 HC + PLE, MTP head, bf16 vision tower; card KL
|
||
0.0789 all-tokens / top-1 92.85 % vs bf16. **MODEL_FILE is the only line
|
||
that moved** — the 2026-10-13 shape (25600/5120 pair, 44928 pool, fp8 KV
|
||
3 × 200,000, MTP 2, wht6, MNBT 2048, froggeric v22.5) carries verbatim.
|
||
Rollback: revert the swap commit (w4dn8 `.rad` stays on disk). The
|
||
fp8-native `.rad` and the raw Swift1.5 BF16 source were DELETED from
|
||
dankserver 2026-10-15 (Frank's one-time rm, volume for this payload;
|
||
rebuild via the fetch + build scripts named in the fp8-native runbook)
|
||
- [[frank-archon-radiance-swift15-w4dn8]] — **RETIRED 2026-10-08 → 10-15
|
||
(superseded by the publisher iq4r swap; payload on disk as the rollback)**:
|
||
Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` on radiance
|
||
1.0.13 (recipe + 10M-token calibration + KLD gate vs the BF16 container);
|
||
the FP8-native payload remains the rollback. **2026-10-13: fp8 KV, and the
|
||
pair re-cut twice** — `--kv-cache-dtype fp8` first at
|
||
`--max-num-seqs 4`/6912 MiB pool (843,566 tokens, slab 22208), then **3 slots
|
||
with the pool cut to the contract (5120 MiB, 608,027 tokens) and the slab
|
||
raised 22208 → 24000 → 25600**, the last 1,600 MiB claimed out of the card's
|
||
settling slack: the engine's ceiling is **30,969 MiB**, not the 29,120 my
|
||
1 GiB reserve invented. fp8 is what funds the slots — the pool must back
|
||
`seqs × window`, and at bf16 four slots cost 11,679 MiB (slab 17,441, −33 %).
|
||
Measured on the slab: A-cold 43.5k prefill 22208 → 3,397, 24000 → 3,290,
|
||
**25600 → 3,584 t/s**; 3-way 197k aggregate 3,051 → **3,589 t/s**. A shipped
|
||
prediction that the slab was a threshold below 26 GiB died on its own
|
||
measurement — it is a slope, and 26,071 (4,379 t/s) is the cliff. The bf16
|
||
arm (2 × 200,000, 462,752 tokens, prefill −29 %, acceptance 0.661 → 0.740)
|
||
and the 4-slot fp8 shape are the documented rollbacks.
|
||
**2026-10-13 also: the pinned-ledger preflight now raises `ttm.pages_limit`
|
||
itself** — the 2026-10-12 power failure
|
||
reset the knob to its default (125,988 MiB) and the endpoint refused to boot
|
||
6,919 times in ~20 h; `scripts/ttm-preflight.sh` is the repair.
|
||
- [[frank-archon-radiance-qwen38-native-fp8]] — **RETIRED 2026-10-08 (the
|
||
w4dn8 swap replaced it; its `.rad` stays on disk as the rollback)**
|
||
(root systemd unit, port 7770): the StillDeadcode **radiance** engine
|
||
(C++/HIP, libr4d gfx1201 kernels, digest-pinned 287 MiB image, engine as
|
||
the image entrypoint) serving the OFFICIAL `Qwen/Qwen3.8-Flash-Next-FP8`
|
||
checkpoint at NATIVE precision — converted no-recipe to
|
||
`qwen4exp-fp8-native.rad` (~161 GiB) after dropping `quantization_config`
|
||
in a staging dir (the plugin descriptor-selection law: nothing claims
|
||
`(qwen4exp, "fp8")`, the shipped plugin claims `(qwen4exp, "")`;
|
||
per-weight encodings are data, not the descriptor) — on BOTH R9700s
|
||
**TP=2** (`--tp-wire wht6` fused all-reduce), experts in a 40 GiB/rank
|
||
pinned host pool + VRAM cache (ratio 0.82 = the measured 99 %-hit point),
|
||
the 47.7 GiB PLE n-gram table PINNED to host RAM (`--ngram-placement
|
||
ram`), fp8 KV 7.32 KiB/token, MTP spec decode depth 3, 2 × 200,000 first
|
||
boot, froggeric v22.5 template, vision + tools + `reasoning_content` auto,
|
||
`/metrics` scraped. Rollback: git tag
|
||
`frank-archon-freetoken-pr587-20261005` (FreeToken image + orcarouter
|
||
payload on disk); older rollback `frank-archon-vllm-shapeG-20261003`
|
||
- [[frank-archon-freetoken-pr587]] — **RETIRED 2026-10-05 (same day;
|
||
rollback target)**: FreeToken at PR #587 head `77a7201` (branch
|
||
rocm714_torch214) built on the host into the local image
|
||
`frank-archon-freetoken:pr587-77a7201` (Dockerfile: digest-pinned
|
||
rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev-symlink law,
|
||
curated --no-deps install per docs/install_amd.md), serving the SAME
|
||
orcarouter FP8 payload on ONE R9700 — experts in pinned host RAM
|
||
(~120 GiB + 47.7 GiB PLE), GPU LRU at 0.10, KV pool 299,968 tokens
|
||
(multiple-of-64 law), 2 slots, parsers AUTO (thinking →
|
||
`reasoning_content`). Measured: boot ~2.3 min, decode 17.1 tok/s
|
||
single-stream, vision "Red" probe passes, native tool_calls; TP=2 does
|
||
not boot on ROCm (pynccl JIT) — second card idle. Superseded by the
|
||
radiance swap for TP=2; the image + payload stay on disk until the
|
||
rollback window closes
|
||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **RETIRED 2026-10-05
|
||
(rollback target; the vLLM trio + env lore live in the tag)**: the
|
||
tcclaviger RDNA4
|
||
vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both
|
||
R9700s, TP=2 with automatic expert offload — `--max-model-len 200704` x 3
|
||
slots, fp8 KV, pool 3,150,000,000 B/rank (measured 398,658 tokens = 1.99x),
|
||
MNBT 1024, gather 32x64, no spec decode, prefetch off, plus
|
||
`--expert-precise-memory` (arena 17.22 GiB / 7,522 slots), GPU_MAX_HW_QUEUES=2
|
||
and a mounted compile cache. Why `--max-num-seqs`
|
||
must stay 3 (SEQS 2 cannot compile this model), why the arena's levers are
|
||
`max_model_len` and the overhead term rather than the KV pool, and the measured envelope
|
||
- [[frank-archon-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
|
||
failed GLM experiment; frank-archon runs Qwen FP8 again, see the intro):
|
||
the tcclaviger RDNA4 vLLM fork
|
||
(digest-pinned `tcclaviger/vllm@sha256:da4029666…`, plain `podman run`,
|
||
no shim) serving Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric
|
||
int4 group 128, ~170 GiB, 34 shards + 2 extras, 113,074 tensors — on BOTH
|
||
R9700s, TP=2, with the MoE experts offloaded to pinned host RAM (12,096
|
||
experts/rank, 73.64 GiB) behind a 14.51 GiB / 2,383-slot VRAM LRU arena; the
|
||
MoE kernel is libr4d's grouped W4A16 GEMM (`R4dW4a16MoEMethod`, reached
|
||
because the checkpoint's `quant_method: auto-round` is overridden to `inc` —
|
||
a compressed-tensors W4A16 checkpoint of the same model has NO offload recipe
|
||
and refuses to serve). 131,072 ctx x 2 seqs, bf16 KV, util 0.98, prefix
|
||
caching, vision and `glm47` tool/reasoning parsers, MTP OFF and prefetch OFF
|
||
by measurement. **Measured decode 9.9-10.0 tok/s, prefill 918 tok/s — the
|
||
35 tok/s target is NOT met, bounded by the fork's expert-gather read path
|
||
(~8.2 GB/s effective vs a 22.74 GB/s link) at a 0.53-0.66 cache hit rate.**
|
||
Rollback: commit `2339fae` (the qwen FP8 + MTP4 shape of 2026-09-29 → 09-30)
|
||
- [[frank-archon-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
|
||
back; payload and R9V state dirs intact on disk, git tag
|
||
frank-archon-r9v-q8-20260929 is the last pre-cutover commit):
|
||
[[frank-archon-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
|
||
(same-day swap to the Q8_0 payload; rollback target, state + payload
|
||
intact on disk): the R9V radiance-vLLM fork serving
|
||
Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) +
|
||
blockfp8 MTP4 (depth 4) + F16 vision on BOTH R9700s — TP=2 tiered
|
||
experts (UVA pinned host copy, 62 hot + 160-slot LRU on rank 0, 428
|
||
static on rank 1), PLE n-gram on ssd-mmap, CED on, 196608 ctx. Cut over
|
||
from llama.cpp the same day; rollback target
|
||
[[frank-archon-llamacpp-qwen38-flash-next-q8]]
|
||
- [[frank-archon-llamacpp-qwen38-flash-next-q8]] — RETIRED 2026-09-27
|
||
(rollback target; the llama.cpp b11030 endpoint of 2026-09-26):
|
||
unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X, bundled runtime)
|
||
serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 across both R9700s —
|
||
routed experts and the 51B-param n-gram table computed in host RAM
|
||
(`-ot exps=CPU,per_layer_token_embd.weight=CPU`), the shared Q8_0 MTP head
|
||
(depth 2) chained behind ngram-mod, BF16 vision, one full 262,144-token
|
||
conversation, `reasoning_content`, the placement law, the cutover and
|
||
rollback
|
||
- [[frank-archon-lucebox-ds4]] — RETIRED 2026-09-26 (same-day swap,
|
||
rollback target; active for hours): the Lucebox DS4.1 engine
|
||
(localhost/lucebox-rocm:gfx1201) serving
|
||
Lucebox/DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB) with streamed experts —
|
||
the fleet's only non-llama.cpp runtime detour
|
||
- [[dankserver-frontier-moe-feasibility]] — VERIFIED 2026-09-28: which of
|
||
DeepSeek-V4.1-Flash (552B), MiMo-V2.6-Flash-MOPD (309B) and GLM-5.3-Flash
|
||
(320B) can run on the 2x R9700 box, on what runtime, and why V4.1 does not
|
||
fit (475 GiB payload, 189 GiB of it Engram; decode is PCIe-expert-stream
|
||
bound, ~23 tok/s ceiling at today's residency). Includes the Affinity
|
||
V4-Flash-to-V4.1 port assessment — reusable expert tiering, four
|
||
architecture gaps, one of them a silent-wrongness trap on the Engram key
|
||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — VERIFIED 2026-09-29:
|
||
the flag-level decode-tuning campaign on the ad hoc qwen-fp8 container
|
||
(tcclaviger RDNA4 fork, orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8,
|
||
TP=2 across both R9700s). **>35 tok/s MET: 37.44-40.57 tok/s with
|
||
`--expert-prefetch` DELETED** — telemetry measured the prefetcher at
|
||
36% precision, wasting ~half the gather traffic (the LRU itself runs at
|
||
82.45% hit). Prefetch-on incumbent 26.8 survived every other challenger
|
||
(geometry, spec depth, TunableOp-corrupts, batch knobs, concurrency,
|
||
ngram-fork-bug); the residual gap to ~60 tok/s is the fork's gather
|
||
read path (6.5-9.5 GB/s effective vs 22.7 at the same slab size)
|
||
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26 (replaced by
|
||
[[frank-archon-lucebox-ds4]], then
|
||
[[frank-archon-llamacpp-qwen38-flash-next-q8]]): vllm-radiance 0.9.3
|
||
serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark
|
||
W4A8) with the z-lab DFlash2-FP8 drafter as ONE tensor-parallel process
|
||
(TP=2) behind :7770, on unquantized bf16 KV (`KV_DTYPE=auto`, cut
|
||
2026-09-21). The FP8 checkpoint is the on-disk rollback. The three-arm
|
||
battery that chose TP=2, the auto-sized KV pool, and the verified
|
||
capability battery (tools, reasoning)
|
||
- [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium:
|
||
the streamlining toolchain (hf-manifest.py, hf-fetch.sh,
|
||
swap-frank-medium.sh), the quantization-aware entrypoint contract, the
|
||
quadlet edit, the deploy, the capability battery, and the rollback path.
|
||
The 2026-09-20 MXFP4 → FP8 swap is the worked example
|
||
- [[frank-medium-llamacpp-dflash2]] — RETIRED 2026-09-16 → 2026-09-20: mainline
|
||
llama.cpp serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the
|
||
DFlash2 Q4_K_M drafter as ONE layer-split process (32:33 over two
|
||
163,840-token slots); superseded by the radiance TP=2 cutover, and the
|
||
rollback target. Keeps the batch/ubatch optimum curve and the per-layer split
|
||
arithmetic
|
||
- [[Dual-R9700-Cooperation]] — whether the two R9700s can serve ONE model: the
|
||
RCCL/PCIe-atomics wall on the old TUF GAMING B650-E, the 2026-09-20 ProArt
|
||
B850-Creator x8/x8 swap that cleared it, and the measured three-way verdict
|
||
(radiance TP=2 95.8 t/s decode vs layer split 45.7 vs llama tensor mode 30.5)
|
||
that put TP=2 in production
|
||
- [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era,
|
||
2026-09-15): charlie12345/ROCmFPX serving
|
||
kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the
|
||
runtime it replaced
|
||
- [[freetoken-frank-small]] — RETIRED 2026-09-15, when the RTX 3090 moved to
|
||
frank-dragoon: the FreeToken v0.1.2 frank-zealot runbook (batsclamp
|
||
Huihui-Qwen3.6-35B-A3B-abliterated-FP8), kept as the retirement record. The
|
||
NAME was reused on 2026-09-16 by a different service on Dankclaw — see
|
||
[[frank-zealot-qwen38-gsq-rco]]
|
||
- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep
|
||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||
- [[deployment]] — deploy and update process for both servers
|
||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||
- [[frank-dragoon-rocmfpx-q6agent-uncensored-mtp]] — **RETIRED 2026-10-03 →
|
||
10-05 (rollback target; payload, fork build and build script on disk)**:
|
||
the pinned ROCmFPX fork serving the uncensored protected-heads
|
||
Q6_0_ROCMFPX_AGENT rung with the repo's MTP-Q4_0 head drafting 4 tokens,
|
||
one 262,144 window, f16 KV; measured 27.6-28.1 t/s prose decode at 86.8 %
|
||
acceptance. Replaced by the halogen DFlash2 endpoint; the ngram-mod-onto-MTP
|
||
rejection (-15 %, recurrent-state rollback) stays a live lesson for this
|
||
architecture.
|
||
- [[frank-dragoon-rocmfpx-q8agent-mtp]] — **RETIRED 2026-10-02 → 10-03 (the
|
||
rollback target; payload still on disk)**: the same pinned ROCmFPX fork
|
||
serving the ALIGNED `Q8_0_ROCMFPX_AGENT` (ftype 115, 17.0 t/s, MTP
|
||
acceptance 0.462) — the MTP-vs-none +123 % A/B, the trap catalogue
|
||
- [[frank-dragoon-gufo-qwen38-27b]] — **RETIRED 2026-10-02 (history; the
|
||
payload dirs were removed from disk 2026-10-03, so this rollback is a
|
||
re-download)** (root systemd unit, port 7770): gufo
|
||
(gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from
|
||
source on Dankclaw, serving unsloth/Qwen3.8-27B-GGUF **UD-Q4_K_XL**
|
||
with the z-lab DFlash2 Q4_K_M drafter — ONE session at the full native
|
||
262,144 window, unquantized f16 KV. Rolled back by Frank after one day
|
||
of [[frank-dragoon-vllm-mimo-v26-bf16]]. The two from-source build
|
||
workarounds, the client-contract delta, the measured prefill/decode
|
||
trade against llama.cpp
|
||
- [[frank-dragoon-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (one day;
|
||
rollback by Frank after the autoresearch loop): the digest-pinned
|
||
kyuz0/vllm-therock-gfx1151 image serving
|
||
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16, 4 x 262144 ctx,
|
||
CROSS-MODEL MTP from Qwen/Qwen3.5-9B — kept for the measured record:
|
||
battery decode 6.19 t/s (MTP-2) vs 9.01 t/s (no draft — spec NET-
|
||
NEGATIVE on real chat+reasoning traffic), the roof math, the greedy-text
|
||
numerics trap, and the death of every ≥6-bit vLLM quant path on gfx1151
|
||
(quark int8 rejected, FP8 kernels CDNA-only, torchao unwired)
|
||
- [[frank-dragoon-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
|
||
[[frank-dragoon-gufo-qwen38-27b]]; rollback target): stock llama.cpp
|
||
(unsloth b11030, ROCm gfx1151) serving
|
||
unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2
|
||
Q4_K_M drafter (depth 4) on the gfx1151 iGPU — one full 262,144-token
|
||
conversation, q8_0 KV (the build lacks fp8), text-only, Qwen-Sharp
|
||
v22.4.0 template, the 114 GiB GTT memory model, client contract,
|
||
troubleshooting
|
||
- [[frank-dragoon-kairic-edge-qwen38-27b]] — RETIRED 2026-09-21 (same-day
|
||
swap, replaced by [[frank-dragoon-unsloth-qwen38-27b-q8xl]]; rollback
|
||
target): the Kairic Edge runtime (ciru-ai/ROCmFPX v1.2, built from
|
||
source) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (IU4 W4A4, 27B dense)
|
||
on the gfx1151 iGPU — one full 262,144-token conversation, MTP depth 4 +
|
||
n-gram drafting, the three PromptForge sidecars, compatibility mode, the
|
||
114 GiB GTT memory model, client contract, troubleshooting
|
||
- [[frank-dragoon-unsloth-qwen38-flash-next]] — RETIRED 2026-09-21 (same-day
|
||
swap, replaced by [[frank-dragoon-kairic-edge-qwen38-27b]]; rollback target):
|
||
the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving
|
||
unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (125B MoE) on the gfx1151 iGPU —
|
||
one 200,000-token conversation, MTP depth 2 via the explicit -md shared-Q8_0
|
||
head, the 114 GiB GTT memory model, client contract, troubleshooting
|
||
- [[frank-dragoon-ciru-orca]] — RETIRED 2026-09-21 (replaced by
|
||
[[frank-dragoon-unsloth-qwen38-flash-next]]; rollback target): the CIRU
|
||
v4.4.1 runtime (private host build, QSA hotfix) serving
|
||
jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU — one
|
||
262,144-token conversation, MTP depth 4, 4096 MiB PLE cache, runtime-root
|
||
gate, client contract, troubleshooting (2026-09-19 → 09-21)
|
||
- [[halogen-flash-server]] — RETIRED 2026-09-19 (replaced by
|
||
[[frank-dragoon-ciru-orca]]; rollback target): peonist-ai halogen-flash-server
|
||
0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b
|
||
.hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
|
||
400,384-position pool, deployed flags, memory model, client contract,
|
||
troubleshooting (2026-09-16 → 09-19)
|
||
- [[frank-zealot-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02** (the same
|
||
rung returned ACTIVE 2026-10-03 at the 2 x 200,704 f16-KV shape —
|
||
[[frank-zealot-llama-mimo-q6k-2x200704]]): llama.cpp (master, CUDA sm_86
|
||
build) serving
|
||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO
|
||
native 262,144-token sessions with ASYMMETRIC KV (K f16 unquantized + V
|
||
q8_0, which forces flash-attn), ~20.9/24.6 GiB measured, text-only (the
|
||
GGUF has no vision), reasoning_content, no spec decode; payload still at
|
||
`/home/dank/models/MiMo-Q6K/`
|
||
- [[frank-zealot-sglang-mimo-v26]] — RETIRED 2026-10-01 (hours; replaced
|
||
by [[frank-zealot-llama-mimo-q6k]] for the asymmetric KV SGLang cannot
|
||
express; venv still on disk, INT8 payload staged to
|
||
`~dank/.AGENTTRASH/20261002-retired-llm-payloads/`): SGLang in a
|
||
self-contained python3.12 venv serving leteschlead MiMo-V2.6 INT8
|
||
W8A16 AutoRound, 2 x 262144 ctx, fp8 KV, in-checkpoint vision,
|
||
83.9 tok/s decode
|
||
- [[frank-zealot-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by
|
||
[[frank-zealot-sglang-mimo-v26]] → the llama.cpp MiMo shape; payload staged
|
||
to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/`):
|
||
mainline llama.cpp serving
|
||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
|
||
3090, with the separate MTP assistant sidecar chained behind ngram-mod,
|
||
q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB.
|
||
- [[frank-zealot-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
|
||
rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
|
||
Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-
|
||
token pool, 2 concurrent full-context sessions, native text+image+audio.
|
||
- [[frank-zealot-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the
|
||
prior frank-zealot runbook (2026-09-16 → 09-17), ISTA-DASLab
|
||
Qwen3.8-27B-GSQ-RCO IQ3_S with the model's own in-file MTP head and
|
||
f16 KV at 131k
|
||
- [[x4-pair-experiment]] — MEASURED 2026-09-16: what mainline llama.cpp can and
|
||
cannot do with the 3090 + Strix Halo pair, over their only link (the 3090's
|
||
Gen4 x4). The tensor inventory that decides any split, the single flag (`-ot`)
|
||
that kills per-tensor placement across devices, the two-device layer splits
|
||
that work (+7 % decode at 64k, cross-vendor included), the harness-parsing trap
|
||
that invalidated the first round, and the iGPU-only result that matches
|
||
halogen's deployed decode rate — at a memory cost the co-tenants cannot pay
|
||
- [[frank-dragoon-ik-uncensored]] — RETIRED 2026-09-16 (rollback target): the
|
||
ik_llama.cpp runbook (root systemd unit, 2026-09-15 → 09-16) serving
|
||
mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090
|
||
(dense/attention/KV) + LPDDR (experts/PLE) hybrid; its measured envelope and
|
||
decode root-cause chain are the record of why the fleet went back to halogen
|
||
- [[dankclaw-iommu-translation-off]] — Dankclaw's IOMMU boot line: **`iommu=pt`**
|
||
since 2026-10-18, measured on this host at no prefill cost (+2.0 %, 1,616 t/s
|
||
cold), which restored the XDNA2 NPU; the `amd_iommu=off` era (2026-09-11) it
|
||
replaced ran on a 13-16 % figure measured upstream only; includes the AMD
|
||
PerfOpt research, what was deliberately not copied from the reference machine,
|
||
and the revert path via `scripts/set-dankclaw-bootline.sh`
|
||
- [[kingjones-strix-rocmfp4]] — PRIOR frank-dragoon runbook and rollback
|
||
target: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params
|
||
(56B MoE + 51.2B PLE), single 113.47 GiB GGUF, on the card's pinned
|
||
kingjones30/ROCmFPX fork (2026-08-30 → 2026-09-11)
|
||
- [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
|
||
- [[frank-medium-batch-ubatch]] — batch/ubatch sweep on the 2x R9700 layer split
|
||
(2026-09-16): the ubatch optimum at 512, why the logical batch is free, and
|
||
the split rebalance that followed it
|
||
- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
|
||
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model
|
||
on Dankclaw (retired)
|
||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on
|
||
Dankserver (frank-zealot, retired 2026-09-02)
|
||
- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for
|
||
frank-zealot (2026-08-26): q8_0 + Hadamard KV cache,
|
||
split-mode none (single-GPU), mandatory FA_ALL_QUANTS build flags
|
||
- [[kingjones-strix-mtp]] — the former Q6 AGENT runbook (restored 2026-09-01
|
||
after Q8_0 failure, then superseded by the STRIX deployment on 2026-08-27)
|
||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime
|
||
on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
|
||
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
|
||
(deployed and reverted 2026-08-22)
|
||
- [[Dankserver System Specs]] — the 2x R9700 box (frank-archon)
|
||
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-dragoon on
|
||
the flash-server v2 runtime since 2026-10-06, frank-zealot on the RTX 3090)
|
||
- [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint
|
||
on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as
|
||
the retirement record — the IOMMU/SVA err=-19 root cause and where
|
||
every dependent route moved
|
||
|
||
## CIRU Reference (Dankclaw)
|
||
|
||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 model, ROCm 7.15 toolchain,
|
||
and build history (on Dankclaw; the slot now serves the kingjones
|
||
ROCmFP4 model, see [[kingjones-strix-mtp]])
|
||
- [[kairic-edge]] — Kairic Edge IU4 runtime: build runbook, binary
|
||
identity, and experiment summary (retired 2026-08-22)
|
||
|
||
## DFlash2 Reference (archived)
|
||
|
||
- [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment
|
||
(2026-08-21)
|
||
|
||
## vLLM Reference (historical)
|
||
|
||
- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
|
||
- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
|
||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware
|
||
notes and sizing (this architecture family is what the retired frank-zealot
|
||
served in the Huihui abliterated FP8 variant — see
|
||
[[freetoken-frank-small]]; the Ornith-1.5-35B-A3B CRACK checkpoint
|
||
that preceded it is archived at [[dealign-ornith-crack]])
|
||
|
||
## Memory Management
|
||
|
||
- [[RAM CPU Offloading in VLLM]] — KV cache swap between GPU VRAM and system RAM
|
||
- [[Hybrid CPU GPU Processing Dankserver]] — NVIDIA GPU compute with system RAM offload
|
||
- [[Shared Memory Architecture Dankclaw]] — AMD unified memory and ROCm configuration
|
||
|
||
## Retired Topology (2026-08-17 redo)
|
||
|
||
- [[Qwen3.8 PP Deployment]] — the retired pipeline-parallel head + worker
|
||
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history (superseded by FastMTP)
|
||
- [[FastFlowLM NPU Inference]] — the retired FastFlowLM NPU endpoint
|
||
- [[frank-dragoon-benchmark]] — backend/spec benchmark behind the old head
|
||
- [[dankserver-vram-growth]] — VRAM/RAM growth behind the retired cuBLAS crash
|
||
- [[DFlash DSpark Speculative Decoding]] — retired DFlash/DSpark drafting
|
||
- [[Quantization by Hardware Architecture]] — which quant format fits which GPU/CPU
|
||
|
||
## Pipeline Parallelism (archive)
|
||
|
||
- [[Pipeline Parallelism/index]] — index of multi-node inference research. The PP
|
||
pair is retired. These pages are research history.
|
||
|
||
## Runtime Research (llama.cpp family)
|
||
|
||
- [[Llama Cpp Runtimes]] — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: positioning, quantization, server fit
|
||
- [[Llama Cpp Code Diff]] — code-level quantization diff of the three runtimes + server recommendations
|
||
- [[SGLang]] — the structured-generation serving engine; when it fits Dankserver (2x R9700) vs Dankclaw (Strix Halo)
|
||
|
||
## Cross-Cutting References (skill distillation, 2026-10-05)
|
||
|
||
One polished reference page per skill-cluster, distilled from the managed
|
||
skill library into the vault (see [log](../log.md) for the wave plan):
|
||
|
||
- [[Quantization Production]] — how we self-make quants per card
|
||
(imatrix/K_P ladders, radiance recipes, REAP/Heretic/abliteration
|
||
prior art), and what each runtime decodes natively
|
||
- [[GGUF Compatibility]] — what a GGUF's tensor histogram and header demand
|
||
of a loader; how the runtime is chosen by the file
|
||
- [[Llama Cpp Serving Sizing]] — slot/pool/KV sizing laws, boot-shape
|
||
laddering, llama-bench discipline
|
||
- [[Llama Cpp Multi-Device Splits]] — layer vs tensor splits, `-ts`/`-ot`
|
||
laws, drafter placement
|
||
- [[PCIe Link and Atomics]] — the link walls, atomics routing, and the
|
||
heterogeneous-pair verdicts
|
||
- [[RDNA4 Container Serving]] — bring-up recipes for RDNA4 containers
|
||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan law, every arena
|
||
lever, and why free VRAM is not arena
|
||
- [[vLLM Boot Forensics and Config Interrogation]] — reading a boot that
|
||
died, and interrogating a running fork's config
|
||
- [[vLLM Porting and Compile Triage]] — porting checkpoints across vLLM
|
||
versions/forks; torch.compile/dynamo breakage triage
|
||
- [[LLM Bench Windows and Measurement]] — window discipline, honest A/B,
|
||
the window script laws
|
||
- [[LLM Performance Diagnosis]] — slow prefill/decode triage on live
|
||
endpoints, IBS attribution, prefix-cache routing
|
||
- [[Model Swap and Rollback]] — the swap procedure, payload preflight,
|
||
research verification, second-endpoint bring-up
|
||
- [[Tcclaviger Fork Ops]] — the fork's sweep harness, measured knob
|
||
verdicts, large-MoE recipe, compile-cache persistence
|
||
- [[Heterogeneous Split Triage]] — triage for unequal-accelerator split
|
||
proposals (the no-go verdict format)
|
||
- [[FreeToken Builds]] — the CUDA and ROCm build paths with their
|
||
dependency traps
|
||
|
||
These sit alongside the operational pages under
|
||
`../Operations/index.md` and `../Agent/index.md`.
|
||
|
||
## Related
|
||
|
||
- [Project knowledge base](../index.md) — the Podman app stacks on the same
|
||
two hosts. OpenWebUI (Dankclaw) consumes the on-host `frank-dragoon`
|
||
endpoint at `:7770`. The Gitea instance on Dankclaw hosts this repository.
|
||
- [Deployment Flow](../Architecture/Deployment%20Flow.md) — the service-scoped
|
||
git-based deploy with inference isolation |