Frank
aaef419a6d
gufo is a vertical inference engine built for exactly this hardware (Ryzen AI MAX+ 395 / Radeon 8060S, gfx1151), and Dankclaw's frank-large has swapped runtimes seven times in search of a better one. This one ships no prebuilt Linux binary, so the runtime is BUILT FROM SOURCE on the host and the payload, port, alias and client contract stay exactly where they are. Runtime: gufo v0.1.1 (b0f8673) at /home/dank/GitPackages/gufo-v0.1.1, built by the new scripts/build-gufo.sh. It links the host ROCm 7.2.4 through its RPATH — the opposite of the unsloth prebuilt it replaces, which carried its own — so two host workarounds are encoded in that script: Ubuntu's librocwmma-dev installs a truncated rocWMMA header set that shadows AMD's and cannot be overridden with -I in HIP mode, and the ROCm clang prefers Ubuntu 26.04's GCC 16 headers over the GCC 15 whose libstdc++ is actually installed. Shape is unchanged: one session, full 262144 ctx, the same UD-Q8_K_XL target and the same DFlash2 drafter, greedy, text-only. Speculation moves from llama.cpp's fixed depth 4 to gufo's adaptive block policy capped at 7 proposals. Client-contract deltas, all documented: the chat template is compiled into the binary (no --chat-template-file, so the Qwen-Sharp 73k-token failure mode is gone), reasoning_effort now takes the full OpenAI set (so models.yml's reasoningEffortMap fold is no longer required — kept anyway, so this commit does not also move every high role from xhigh to high effort), and there is no KV-quant or prompt-cache-RAM flag. The test seam now runs the serve script against a stub engine and asserts on the argv it actually execs, instead of grepping the script's text. The deploy handler gates on the runtime binary and returns before it stops the serving unit. Measured trade, from gufo's own Q8_K_XL tables against llama.cpp b11069: prefill +37% to +80%, cold load 4.5x faster, decode +7% at zero depth — but -14% at 65,536 tokens of depth and -24% at 131,072. On the 8-bit target the crossover is ~32k, and frank-large runs one 262,144 conversation. The Q4_K_XL target shows no such regression.
299 lines
17 KiB
Markdown
299 lines
17 KiB
Markdown
# LLM Fleet — Knowledge Base
|
||
|
||
Documentation for the LLM inference fleet on the two hosts (2026-09-27):
|
||
**Dankserver runs R9V radiance-vLLM on frank-primary** (the drwolfen fork
|
||
serving the unsloth Qwen3.8-Flash-Next Q8_0 + MTP4 across both Radeon AI
|
||
PRO R9700s on :7770, docker-pinned runtime driven through a docker→podman
|
||
shim, the Qwen-Sharp template, 131072 ctx), and Dankclaw runs two llama.cpp endpoints —
|
||
stock llama.cpp (unsloth b11030, ROCm gfx1151; frank-large, unsloth
|
||
Qwen3.8-27B UD-Q8_K_XL + the z-lab DFlash2 Q4_K_M drafter on the gfx1151
|
||
iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B
|
||
QAT Q4_K_M with MTP and vision on the RTX 3090, port 7771).
|
||
|
||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||
> `serverAppConfigs-docs/` root to `serverAppConfigs-docs/LLM/`. The fleet scripts and units now live in the
|
||
> repo's `scripts/` directory.
|
||
|
||
## Deployment Topology
|
||
|
||
```mermaid
|
||
graph LR
|
||
subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
|
||
A[2x Radeon AI PRO R9700
|
||
gfx1201, 32 GiB each]
|
||
B2[R9V radiance-vLLM fork, docker-pinned
|
||
via docker-podman shim
|
||
frank-primary, port 7770]
|
||
C[Qwen3.8-Flash-Next Q8_0
|
||
TP=2 tiered experts, MTP4, sharp template, 131072 ctx, vision]
|
||
end
|
||
|
||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||
E2[RTX 3090 (PCIe x4 dock) +
|
||
gfx1151 iGPU 114 GiB GTT]
|
||
E[stock llama.cpp (unsloth b11030, gfx1151)
|
||
frank-large, port 7770]
|
||
F[Qwen3.8-27B UD-Q8_K_XL + DFlash2 Q4_K_M drafter
|
||
262144 ctx / 1 slot, DFlash2 depth 4, q8_0 KV]
|
||
G[mainline llama.cpp 37b53fd
|
||
frank-small, port 7771]
|
||
H[Gemma4-12B QAT Q4_K_M
|
||
4 x 262144 ctx, MTP + vision + audio]
|
||
E2 --> E --> F
|
||
E2 --> G --> H
|
||
end
|
||
|
||
CLI[LLM Clients / API] -->|HTTP :7770| B2
|
||
CLI -->|HTTP :7770| E
|
||
CLI -->|HTTP :7771| G
|
||
```
|
||
|
||
## Servers
|
||
|
||
| Server | GPU | VRAM / Memory | OS |
|
||
| --------------------------------------- | ----------------------------- | -------------------- | ------------ |
|
||
| [[Dankserver System Specs\|Dankserver]] | 2x AMD Radeon AI PRO R9700 | 2x 32 GB VRAM, 246 GB RAM | Ubuntu 26.04 |
|
||
| [[Dankclaw System Specs\|Dankclaw]] | AMD Radeon 8060S (Strix Halo) | gfx1151 unified memory | Ubuntu 26.04 |
|
||
|
||
## Active Deployment
|
||
|
||
- [[frank-primary-r9v-radiance-qwen38-q8]] — the ACTIVE frank-primary
|
||
runbook (2026-09-27, root systemd unit, port 7770): the R9V
|
||
radiance-vLLM fork (drwolfen/radiance-vllm-qwen4exp, docker-pinned
|
||
image bundle `2dac17a2`, driven through a docker→podman shim) serving
|
||
the unsloth Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 shards) +
|
||
blockfp8 MTP4 (depth 4) + vision on BOTH R9700s — TP=2 tiered experts
|
||
(80-slot LRU cache), PLE n-gram PINNED in host RAM (54.4 GB), CED off,
|
||
131072 ctx, the Qwen-Sharp v22.4.0 template via the launcher's
|
||
`R9V_CHAT_TEMPLATE_FILE` passthrough (fork `30d4a32`), the quantized
|
||
inject-shard dev-overlay adapter (fork `363d606`), reasoning channel
|
||
`reasoning`, parser `qwen3_coder`. The tuned shape lives in the
|
||
on-host q8xl setup state. Rollback:
|
||
[[frank-primary-r9v-radiance-qwen38-uncensored]]
|
||
- [[frank-primary-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
|
||
(same-day swap to the Q8_0 payload; rollback target, state + payload
|
||
intact on disk): the R9V radiance-vLLM fork serving
|
||
Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) +
|
||
blockfp8 MTP4 (depth 4) + F16 vision on BOTH R9700s — TP=2 tiered
|
||
experts (UVA pinned host copy, 62 hot + 160-slot LRU on rank 0, 428
|
||
static on rank 1), PLE n-gram on ssd-mmap, CED on, 196608 ctx. Cut over
|
||
from llama.cpp the same day; rollback target
|
||
[[frank-primary-llamacpp-qwen38-flash-next-q8]]
|
||
- [[frank-primary-llamacpp-qwen38-flash-next-q8]] — RETIRED 2026-09-27
|
||
(rollback target; the llama.cpp b11030 endpoint of 2026-09-26):
|
||
unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X, bundled runtime)
|
||
serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 across both R9700s —
|
||
routed experts and the 51B-param n-gram table computed in host RAM
|
||
(`-ot exps=CPU,per_layer_token_embd.weight=CPU`), the shared Q8_0 MTP head
|
||
(depth 2) chained behind ngram-mod, BF16 vision, one full 262,144-token
|
||
conversation, `reasoning_content`, the placement law, the cutover and
|
||
rollback
|
||
- [[frank-primary-lucebox-ds4]] — RETIRED 2026-09-26 (same-day swap,
|
||
rollback target; active for hours): the Lucebox DS4.1 engine
|
||
(localhost/lucebox-rocm:gfx1201) serving
|
||
Lucebox/DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB) with streamed experts —
|
||
the fleet's only non-llama.cpp runtime detour
|
||
- [[dankserver-frontier-moe-feasibility]] — VERIFIED 2026-09-28: which of
|
||
DeepSeek-V4.1-Flash (552B), MiMo-V2.6-Flash-MOPD (309B) and GLM-5.3-Flash
|
||
(320B) can run on the 2x R9700 box, on what runtime, and why V4.1 does not
|
||
fit (475 GiB payload, 189 GiB of it Engram; decode is PCIe-expert-stream
|
||
bound, ~23 tok/s ceiling at today's residency). Includes the Affinity
|
||
V4-Flash-to-V4.1 port assessment — reusable expert tiering, four
|
||
architecture gaps, one of them a silent-wrongness trap on the Engram key
|
||
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26 (replaced by
|
||
[[frank-primary-lucebox-ds4]], then
|
||
[[frank-primary-llamacpp-qwen38-flash-next-q8]]): vllm-radiance 0.9.3
|
||
serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark
|
||
W4A8) with the z-lab DFlash2-FP8 drafter as ONE tensor-parallel process
|
||
(TP=2) behind :7770, on unquantized bf16 KV (`KV_DTYPE=auto`, cut
|
||
2026-09-21). The FP8 checkpoint is the on-disk rollback. The three-arm
|
||
battery that chose TP=2, the auto-sized KV pool, and the verified
|
||
capability battery (tools, reasoning)
|
||
- [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium:
|
||
the streamlining toolchain (hf-manifest.py, hf-fetch.sh,
|
||
swap-frank-medium.sh), the quantization-aware entrypoint contract, the
|
||
quadlet edit, the deploy, the capability battery, and the rollback path.
|
||
The 2026-09-20 MXFP4 → FP8 swap is the worked example
|
||
- [[frank-medium-llamacpp-dflash2]] — RETIRED 2026-09-16 → 2026-09-20: mainline
|
||
llama.cpp serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the
|
||
DFlash2 Q4_K_M drafter as ONE layer-split process (32:33 over two
|
||
163,840-token slots); superseded by the radiance TP=2 cutover, and the
|
||
rollback target. Keeps the batch/ubatch optimum curve and the per-layer split
|
||
arithmetic
|
||
- [[Dual-R9700-Cooperation]] — whether the two R9700s can serve ONE model: the
|
||
RCCL/PCIe-atomics wall on the old TUF GAMING B650-E, the 2026-09-20 ProArt
|
||
B850-Creator x8/x8 swap that cleared it, and the measured three-way verdict
|
||
(radiance TP=2 95.8 t/s decode vs layer split 45.7 vs llama tensor mode 30.5)
|
||
that put TP=2 in production
|
||
- [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era,
|
||
2026-09-15): charlie12345/ROCmFPX serving
|
||
kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the
|
||
runtime it replaced
|
||
- [[freetoken-frank-small]] — RETIRED 2026-09-15, when the RTX 3090 moved to
|
||
frank-large: the FreeToken v0.1.2 frank-small runbook (batsclamp
|
||
Huihui-Qwen3.6-35B-A3B-abliterated-FP8), kept as the retirement record. The
|
||
NAME was reused on 2026-09-16 by a different service on Dankclaw — see
|
||
[[frank-small-qwen38-gsq-rco]]
|
||
- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep
|
||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||
- [[deployment]] — deploy and update process for both servers
|
||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||
- [[frank-large-gufo-qwen38-27b]] — ACTIVE frank-large runbook since
|
||
2026-09-28 (root systemd unit, port 7770): gufo (gufo-org/gufo) v0.1.1, the
|
||
Strix Halo inference engine, built from source on Dankclaw, serving
|
||
unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense) with the z-lab DFlash2
|
||
Q4_K_M drafter (adaptive block policy) on the gfx1151 iGPU — one full
|
||
262,144-token conversation, greedy, text-only, the two from-source build
|
||
workarounds, the client-contract delta, the measured prefill/decode trade
|
||
against llama.cpp, troubleshooting
|
||
- [[frank-large-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
|
||
[[frank-large-gufo-qwen38-27b]]; rollback target): stock llama.cpp
|
||
(unsloth b11030, ROCm gfx1151) serving
|
||
unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2
|
||
Q4_K_M drafter (depth 4) on the gfx1151 iGPU — one full 262,144-token
|
||
conversation, q8_0 KV (the build lacks fp8), text-only, Qwen-Sharp
|
||
v22.4.0 template, the 114 GiB GTT memory model, client contract,
|
||
troubleshooting
|
||
- [[frank-large-kairic-edge-qwen38-27b]] — RETIRED 2026-09-21 (same-day
|
||
swap, replaced by [[frank-large-unsloth-qwen38-27b-q8xl]]; rollback
|
||
target): the Kairic Edge runtime (ciru-ai/ROCmFPX v1.2, built from
|
||
source) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (IU4 W4A4, 27B dense)
|
||
on the gfx1151 iGPU — one full 262,144-token conversation, MTP depth 4 +
|
||
n-gram drafting, the three PromptForge sidecars, compatibility mode, the
|
||
114 GiB GTT memory model, client contract, troubleshooting
|
||
- [[frank-large-unsloth-qwen38-flash-next]] — RETIRED 2026-09-21 (same-day
|
||
swap, replaced by [[frank-large-kairic-edge-qwen38-27b]]; rollback target):
|
||
the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving
|
||
unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (125B MoE) on the gfx1151 iGPU —
|
||
one 200,000-token conversation, MTP depth 2 via the explicit -md shared-Q8_0
|
||
head, the 114 GiB GTT memory model, client contract, troubleshooting
|
||
- [[frank-large-ciru-orca]] — RETIRED 2026-09-21 (replaced by
|
||
[[frank-large-unsloth-qwen38-flash-next]]; rollback target): the CIRU
|
||
v4.4.1 runtime (private host build, QSA hotfix) serving
|
||
jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU — one
|
||
262,144-token conversation, MTP depth 4, 4096 MiB PLE cache, runtime-root
|
||
gate, client contract, troubleshooting (2026-09-19 → 09-21)
|
||
- [[halogen-flash-server]] — RETIRED 2026-09-19 (replaced by
|
||
[[frank-large-ciru-orca]]; rollback target): peonist-ai halogen-flash-server
|
||
0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b
|
||
.hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
|
||
400,384-position pool, deployed flags, memory model, client contract,
|
||
troubleshooting (2026-09-16 → 09-19)
|
||
- [[frank-small-gemma4-12b]] — ACTIVE frank-small runbook since 2026-09-17
|
||
(root systemd unit, port 7771): mainline llama.cpp serving
|
||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
|
||
3090, with the separate MTP assistant sidecar chained behind ngram-mod,
|
||
q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB. A same-day
|
||
vLLM detour (2026-09-19) was reverted the same day; see below.
|
||
- [[frank-small-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
|
||
rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
|
||
Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-
|
||
token pool, 2 concurrent full-context sessions, native text+image+audio.
|
||
- [[frank-small-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the
|
||
prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab
|
||
Qwen3.8-27B-GSQ-RCO IQ3_S with the model's own in-file MTP head and
|
||
f16 KV at 131k
|
||
- [[x4-pair-experiment]] — MEASURED 2026-09-16: what mainline llama.cpp can and
|
||
cannot do with the 3090 + Strix Halo pair, over their only link (the 3090's
|
||
Gen4 x4). The tensor inventory that decides any split, the single flag (`-ot`)
|
||
that kills per-tensor placement across devices, the two-device layer splits
|
||
that work (+7 % decode at 64k, cross-vendor included), the harness-parsing trap
|
||
that invalidated the first round, and the iGPU-only result that matches
|
||
halogen's deployed decode rate — at a memory cost the co-tenants cannot pay
|
||
- [[frank-large-ik-uncensored]] — RETIRED 2026-09-16 (rollback target): the
|
||
ik_llama.cpp runbook (root systemd unit, 2026-09-15 → 09-16) serving
|
||
mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090
|
||
(dense/attention/KV) + LPDDR (experts/PLE) hybrid; its measured envelope and
|
||
decode root-cause chain are the record of why the fleet went back to halogen
|
||
- [[dankclaw-iommu-translation-off]] — why Dankclaw boots with
|
||
`amd_iommu=off` since 2026-09-11 (13-16 % prefill, measured upstream),
|
||
what it cost (the NPU endpoint), what was deliberately not copied from
|
||
the reference machine, and how to revert
|
||
- [[kingjones-strix-rocmfp4]] — PRIOR frank-large runbook and rollback
|
||
target: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params
|
||
(56B MoE + 51.2B PLE), single 113.47 GiB GGUF, on the card's pinned
|
||
kingjones30/ROCmFPX fork (2026-08-30 → 2026-09-11)
|
||
- [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
|
||
- [[frank-medium-batch-ubatch]] — batch/ubatch sweep on the 2x R9700 layer split
|
||
(2026-09-16): the ubatch optimum at 512, why the logical batch is free, and
|
||
the split rebalance that followed it
|
||
- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
|
||
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model
|
||
on Dankclaw (retired)
|
||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on
|
||
Dankserver (frank-small, retired 2026-09-02)
|
||
- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for
|
||
frank-small (2026-08-26): q8_0 + Hadamard KV cache,
|
||
split-mode none (single-GPU), mandatory FA_ALL_QUANTS build flags
|
||
- [[kingjones-strix-mtp]] — the former Q6 AGENT runbook (restored 2026-09-01
|
||
after Q8_0 failure, then superseded by the STRIX deployment on 2026-08-27)
|
||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime
|
||
on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
|
||
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
|
||
(deployed and reverted 2026-08-22)
|
||
- [[Dankserver System Specs]] — the 2x R9700 box (frank-primary)
|
||
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on
|
||
the unsloth b11030 runtime since 2026-09-21, frank-small on the RTX 3090)
|
||
- [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint
|
||
on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as
|
||
the retirement record — the IOMMU/SVA err=-19 root cause and where
|
||
every dependent route moved
|
||
|
||
## CIRU Reference (Dankclaw)
|
||
|
||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 model, ROCm 7.15 toolchain,
|
||
and build history (on Dankclaw; the slot now serves the kingjones
|
||
ROCmFP4 model, see [[kingjones-strix-mtp]])
|
||
- [[kairic-edge]] — Kairic Edge IU4 runtime: build runbook, binary
|
||
identity, and experiment summary (retired 2026-08-22)
|
||
|
||
## DFlash2 Reference (archived)
|
||
|
||
- [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment
|
||
(2026-08-21)
|
||
|
||
## vLLM Reference (historical)
|
||
|
||
- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
|
||
- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
|
||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware
|
||
notes and sizing (this architecture family is what the retired frank-small
|
||
served in the Huihui abliterated FP8 variant — see
|
||
[[freetoken-frank-small]]; the Ornith-1.5-35B-A3B CRACK checkpoint
|
||
that preceded it is archived at [[dealign-ornith-crack]])
|
||
|
||
## Memory Management
|
||
|
||
- [[RAM CPU Offloading in VLLM]] — KV cache swap between GPU VRAM and system RAM
|
||
- [[Hybrid CPU GPU Processing Dankserver]] — NVIDIA GPU compute with system RAM offload
|
||
- [[Shared Memory Architecture Dankclaw]] — AMD unified memory and ROCm configuration
|
||
|
||
## Retired Topology (2026-08-17 redo)
|
||
|
||
- [[Qwen3.8 PP Deployment]] — the retired pipeline-parallel head + worker
|
||
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history (superseded by FastMTP)
|
||
- [[FastFlowLM NPU Inference]] — the retired FastFlowLM NPU endpoint
|
||
- [[frank-large-benchmark]] — backend/spec benchmark behind the old head
|
||
- [[dankserver-vram-growth]] — VRAM/RAM growth behind the retired cuBLAS crash
|
||
- [[DFlash DSpark Speculative Decoding]] — retired DFlash/DSpark drafting
|
||
- [[Quantization by Hardware Architecture]] — which quant format fits which GPU/CPU
|
||
|
||
## Pipeline Parallelism (archive)
|
||
|
||
- [[Pipeline Parallelism/index]] — index of multi-node inference research. The PP
|
||
pair is retired. These pages are research history.
|
||
|
||
## Runtime Research (llama.cpp family)
|
||
|
||
- [[Llama Cpp Runtimes]] — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: positioning, quantization, server fit
|
||
- [[Llama Cpp Code Diff]] — code-level quantization diff of the three runtimes + server recommendations
|
||
- [[SGLang]] — the structured-generation serving engine; when it fits Dankserver (2x R9700) vs Dankclaw (Strix Halo)
|
||
|
||
## Related
|
||
|
||
- [Project knowledge base](../index.md) — the Podman app stacks on the same
|
||
two hosts. OpenWebUI (Dankclaw) consumes the on-host `frank-large`
|
||
endpoint at `:7770`. The Gitea instance on Dankclaw hosts this repository.
|
||
- [Deployment Flow](../Architecture/Deployment%20Flow.md) — the service-scoped
|
||
git-based deploy with inference isolation |