diff --git a/AGENTS.md b/AGENTS.md index 1b83271..2b331d7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -52,6 +52,32 @@ Retired scripts live in `old/` (reference only) and `.AGENTTRASH/`. - `docs/` — knowledge base (Obsidian vault). `docs/LLM/` is the LLM fleet - `tests/` — bats test suite +## Documentation Authoring + +The `docs/` vault is the project knowledge base (Obsidian, wikilinks) and is an +OKF (Open Knowledge Format) knowledge bundle. Follow the OKF spec for every doc +you create or edit. + +- **Canonical spec:** `~/.pi/agent/docs/references/okf-spec.md` — the OKF v0.1 + spec in the **system-config knowledge base** at `~/.pi/agent/docs`. The repo's + own `docs/references/okf-spec.md` is deprecated/removed; use the system one. The + spec index and maintenance guide are + `~/.pi/agent/docs/references/index.md` and + `~/.pi/agent/docs/references/okf-maintenance.md`. +- **Read before you touch:** open the relevant `index.md` first + (`docs/index.md` → `docs/LLM/index.md`). Read the whole target file before + editing it. +- **One concept per file**, with YAML frontmatter that has a non-empty `type:` + field (`reference` / `guide` / `configuration` / `research`) plus `description`, + `tags`, and `timestamp`. Split broad topics; keep pages ~1–500 lines. +- **Progressive disclosure:** an `index.md` at each level links its children. + Every `docs/**/*.md` (except `index.md`) must be linked from + `docs/index.md` with `[[Title]]` — the docs-completeness test checks this. +- **Cross-link** siblings as `[[Title]]` wikilinks; cite external claims under a + `# Citations` section. +- **Always update docs** when a task changes behavior, then run + `./tests/run.sh` (the docs-completeness seam is in there) before commit. + ## Git Workflow All edits flow through git. Do not edit files on the remote machine directly. diff --git a/docs/LLM/index.md b/docs/LLM/index.md index 47a9e7e..b4ebf7d 100644 --- a/docs/LLM/index.md +++ b/docs/LLM/index.md @@ -109,6 +109,12 @@ MTP draft + Turbo4 V cache, 262K ctx / 1 slot] - [[Pipeline Parallelism/index]] — index of multi-node inference research. The PP pair is retired. These pages are research history. +## Runtime Research (llama.cpp family) + +- [[Llama Cpp Runtimes]] — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: positioning, quantization, server fit +- [[Llama Cpp Code Diff]] — code-level quantization diff of the three runtimes + server recommendations +- [[SGLang]] — the structured-generation serving engine; when it fits Dankserver (3090) vs Dankclaw (Strix Halo) + ## Related - [Project knowledge base](../index.md) — the Podman app stacks on the same diff --git a/docs/Research/Llama Cpp Code Diff.md b/docs/Research/Llama Cpp Code Diff.md new file mode 100644 index 0000000..18aa92b --- /dev/null +++ b/docs/Research/Llama Cpp Code Diff.md @@ -0,0 +1,133 @@ +--- +type: research +title: Llama.cpp Code Diff +description: Code-level diff of mainline llama.cpp, charlie12345/ROCmFPX, and ik_llama.cpp with a focus on quantization code, ggml type numbers, and the differences that matter for Dankserver (RTX 3090) and Dankclaw (Strix Halo). +tags: + - llama.cpp + - quantization + - ggml + - ROCmFPX + - ik_llama + - code-diff +timestamp: 2026-09-01 +--- + +# Llama.cpp Code Diff + +Code-level diff of the three runtimes, focused on **quantization code** and the +differences that affect our two servers. Positioning overview is in +[[Llama Cpp Runtimes]]. + +## 1. Quantization file map + +Where quant lives in each runtime (paths relative to the repo root): + +| File / path | Mainline | ROCmFPX | ik_llama.cpp | +|-------------|----------|---------|--------------| +| `ggml/include/ggml.h` (`enum ggml_type`) | FP16/FP32/BF16, legacy `Q*_0`, K-quant types, `IQ*_` | **adds** `Q4_0_ROCMFP4` … `Q2_0_ROCMFPX`, `Q4_0_ROCMI4`, `Turbo3_0`, `Turbo4_0` (enums **100–108**) | IQK non-linear (`IQ4_NL` etc.), Trellis `IQ*_KT`, MXFP4, Bitnet, `Q8_KV` | +| `ggml/src/ggml-quants.*` | K/I/legacy dequant, row-interleaved 3-bit | `block_rocmfp4` unpack + E4M3 scaling, IU4 WMMA dispatch | `iqk/*` fused quant-gemms, IQK row-interleave, Trellis decode | +| `tools/quantize/quantize.c` | all mainline quants + `--imatrix` | ROCmFPX quantizer (`STRIX`/`FAST`/`_AGENT` presets) | IQK/Trellis/MXFP4 packers, `--cache-type-k/v` + Hadamard | +| `gguf.cpp` / `gguf-io.c` | standard ftype table | **custom ftypes** 100–106 (FP4), 110–113 (FPX), 118 (I4) | IQK/Trellis ftype tags | +| KV cache | `f32/f16/bf16/q8_0/q4_0/iq4_nl/q5_0/q5_1` (needs FlashAttention) | **TurboQuant** `Turbo3_0`/`Turbo4_0`/`V` | `Q8_KV`, `--k-cache-hadamard`, `--v-cache-hadamard` | + +Two number spaces in ROCmFPX (authoritative `ggml/include/ggml.h`, sha +`c49ebdbd`): +- **`ggml_type` enum** (tensor storage): `Q4_0_ROCMFP4`=100, `Q4_0_ROCMFP4_FAST`=101, + `Q6_0_ROCMFPX`=102, `Q8_0_ROCMFPX`=103, `Q3_0_ROCMFPX`=104, `Turbo3_0`=105, + `Turbo4_0`=106, `Q2_0_ROCMFPX`=107, `Q4_0_ROCMI4`=108, count 109. +- **`ggml_ftype`** (GGUF model-file): ROCmFP4 family `100–106` (`STRIX`=105, + `STRIX_LEAN`=106), ROCmFPX family `110–113` (FP6=110, FP8=111, FP3=112, + FP2=113), `ROCmI4`=118. + +## 2. Comparison table + +| Runtime | Quant formats | Granularity | Custom ggml types | MoE | Attention accel | Hybrid GPU/CPU offload | Build backends | +|---------|---------------|-------------|-------------------|-----|-----------------|------------------------|----------------| +| Mainline | K/I/legacy/TQ, imatrix, row-RI 3-bit | grouped/per-block + double-quant | none | native `--cpu-moe` | FlashAttention (multi-backend) | `--n-gpu-layers`, partial offload | CPU/CUDA/Vulkan/Metal/SYCL/HIP-ROCm/CANN | +| charlie12345/ROCmFPX | ROCmFP2/3/4/6/8, ROCmI4, TurboQuant KV | block + E4M3 codebook | **yes (100–108)** | MTP draft head (bundled) | **MTEA** + TurboQuant FA (custom) | Strix-tuned MTP profile | HIP/ROCm + Vulkan (**CUDA off**), CPU ref | +| ik_llama.cpp | IQ* non-linear, improved K, IQK row-RI, Trellis, MXFP4, Q8_KV, Bitnet | grouped + non-linear + row-interleaved | IQK/Trellis/MXFP4/Bitnet tags | **fused MoE**, `-ooae`, `--fit`, `-ot`, GDN | **FlashMLA** (DeepSeek MLA) | **finest-grained** override-tensor / cpu-moe / fit | CPU (AVX-512/VNNI), CUDA (Turing+), Vulkan, ROCm(weak) | + +## 3. Code-difference narrative + +**What each fork ADDED that mainline lacks:** +- **ROCmFPX** — ~10 new `ggml_type` entries, the `block_rocmfp4` packed-nibble + format, `*_ROCMFPX`/`*_ROCMI4` ftypes, tensor-aware presets (`*_AGENT`, + `*_COHERENT`, `*_STRIX`), `block_rocmfp4` dequant + dispatch kernels, + TurboQuant KV types, MTEA attention kernel. Mainline does not know these + tensor types. It is a *fork* that adds a new backend: custom tensor + registration + HIP/ROCm + Vulkan kernels + CPU reference. **CUDA is disabled.** +- **ik_llama.cpp** — new code under `ggml/src/iqk/` (IQK GEMM + row-interleave), + CUDA IQK flash-attn kernels (`GGML_IQK_FA_ALL_QUANTS=ON`), fused MoE + (`fnn_up`+`fnn_gate`), fused delta-net (GDN) SSM ops, `flash_fwd_mla` for + DeepSeek MLA/DSA, and hybrid-offload controls (`-ot/--override-tensor`, + `--cpu-moe`, `-ooae/--offload-only-active-experts`, `-ser/--smart-expert-reduction`, + `--fit`, `-op/--offload-policy`, split-mode graph). Run-time quant repack + (`-rtr`). **Additive** — GGUF remains compatible. + +**What mainline has that the forks lag on:** +- **ik_llama.cpp**: architecture baseline is **Aug-2024** — newest + architectures arrive late (PRs move fast, but the core is ~1.5y behind). +- **ROCmFPX**: **AMD-first, experimental**; narrower, must rebuild/re-quant per + fork line, CUDA ecosystem (the fleet's 3090) is unreachable. + +**The one caveat that changes a build (ik_llama.cpp + `-rtr`):** +K-quants have **no CUDA row-interleaved implementation**. With `-rtr`, every +CPU-resident K-quant tensor is repacked to row-interleaved and then **run on CPU** +even where GPU would be faster → lower prompt-processing speed. For a MoE that +keeps experts on CPU (`--cpu-moe`), this is a real hit. Prefer IQK/R4/Trellis/ +MXFP4 variants, or skip `-rtr` and place tensors with `--fit`/`-ot`. + +## 4. Server-specific benefits + +### Dankserver — NVIDIA RTX 3090 (24 GB), 62 GB RAM — 35B MoE (Ornith-1.5-35B-A3B CRACK) +- The winning lever for a **low-VRAM CUDA MoE** is **expert-aware offload + fused + MoE**, which is ik_llama.cpp's core strength: + - `-ooae/--offload-only-active-experts` — with ~8 active experts this cuts + RAM→VRAM traffic dramatically vs shipping all experts. + - `-fmoe/--fused-moe`, GDN fusion (the CRACK model is a hybrid GatedDeltaNet + + MoE), `--fit`, `-ot/--override-tensor`, `--cpu-moe`. +- **Quant pick for the 3090:** `Q4_K_M`/`Q5_K_M` on mainline, or an **IQK/R4 / + Trellis / MXFP4** variant on ik_llama.cpp that has a **CUDA** path. Avoid raw + K-quant + `-rtr`. Enable FlashAttention; `GGML_CUDA_FORCE_MMQ=1`; + `ngram-mod` drafts (MoEs want long drafts). +- ⚠️ **Expert-count discrepancy:** the CRACK GGUF README says **256 experts / 8 + active**; the fleet brief cites ~36. Verify `n_expert` / `n_expert_used` from + the GGUF before finalizing the offload recipe — both point at expert-aware + offload being the right tool. +- **Mainline only?** Fine: `--cpu-moe` + `Q4_K_M` + FlashAttention + `ngram-mod`. +- **ROCmFPX?** No — ROCm-only formats, CUDA disabled on the builds. + +### Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified), ROCm 7.2.4 — Qwen3.8-27B ROCmFP4 STRIX MTP +- **ROCmFPX is the natural fit** and stays the choice: native IU4 (ROCmI4), + ROCmFP4 weight formats, MTP draft head, TurboQuant KV, 262K / 1 slot. The + production `build-promptforge` `llama-server` already runs there. +- **ik_llama.cpp on Strix?** ROCm is its **weakest** backend (gfx1151 not + validated, k-quant-no-hip-row-interleave), and we'd lose the bespoke MTP + draft-head speedup. Higher-risk change; only with a small soak-test first. +- **Mainline on Strix** works for correctness with standard quants but loses FPX, + IU4, TurboQuant, and MTEA. + +## 5. Recommendations for our fleet + +- **Dankclaw:** keep **charlie12345/ROCmFPX** (already production-tuned for + gfx1151). If you want more quant variety on Strix, explore the ROCmFP2 (`Q2_0_ROCMFPX`, + 2.5 bpw, S40 codebook) preview — but do not port to ik_llama.cpp as primary. +- **Dankserver:** **try ik_llama.cpp** (CUDA-90 + AVX-512 build) for fused-MoE + + `offload-only-active-experts`, staying off raw K-quant `-rtr`. If that is too + much surface, stay on **mainline** with `--cpu-moe` + `Q4_K_M` + FlashAttention + + `ngram-mod`. Neither box needs ROCmFP4. +- **Common:** pin a commit per fleet line; re-test after any quant/KV/context + change (FlashAttention + KV-quant are coupled). + +## See Also +- [[Llama Cpp Runtimes]] — positioning + quantization overview +- [[dealign-ornith-crack]] — the CRACK MoE on Dankserver +- [[kingjones-strix-mtp]] — the ROCmFP4 model on Dankclaw +- [[Quantization by Hardware Architecture]] — quant vs hardware + +## Citations +- Mainline quant + backends — https://github.com/ggml-org/llama.cpp · + tools/quantize/README.md · docs/speculative.md +- ROCmFPX enum (ggml_type 100–108) — https://github.com/charlie12345/ROCmFPX/blob/main/ggml/include/ggml.h +- ik_llama.cpp quant/MoE/FlashMLA/backends — https://github.com/ikawrakow/ik_llama.cpp (README, docs/parameters.md, docs/build.md) · DeepWiki · discussions#562 +- ik_llama.cpp `-rtr` K-quant trap & backend-bandwidth caveat — https://github.com/ikawrakow/ik_llama.cpp/blob/main/README.md diff --git a/docs/Research/Llama Cpp Runtimes.md b/docs/Research/Llama Cpp Runtimes.md new file mode 100644 index 0000000..7dca3d7 --- /dev/null +++ b/docs/Research/Llama Cpp Runtimes.md @@ -0,0 +1,141 @@ +--- +type: research +title: Llama.cpp Runtimes +description: Compared mainline llama.cpp, charlie12345/ROCmFPX, and ik_llama.cpp — positioning, quantization formats, code-level differences, and fit for Dankserver (RTX 3090) and Dankclaw (Strix Halo). +tags: + - llama.cpp + - quantization + - ROCmFPX + - ik_llama + - inference + - amd + - nvidia +timestamp: 2026-09-01 +--- + +# Llama.cpp Runtimes + +Comparison of three `llama.cpp`-derived inference runtimes used (or usable) on +the two homelab boxes: **mainline** `ggml-org/llama.cpp`, +**charlie12345/ROCmFPX** (what Dankclaw runs), and **ik_llama.cpp** +(ikawrakow). The goal is to know, for each server, which quant formats are on +the table and which runtime best fits the hardware. + +Full code-level quantification is in [[Llama Cpp Code Diff]]. This page is the +positioning + quantization summary. + +## Which runtime is which + +| Runtime | Repo | What it is | Primary backends | For our boxes | +|---------|------|-----------|------------------|---------------| +| Mainline | [`ggml-org/llama.cpp`](https://github.com/ggml-org/llama.cpp) | Canonical, general-purpose, one codebase for every backend | CPU, CUDA, Vulkan, Metal, SYCL, HIP/ROCm, CANN, OpenVINO | Baseline both boxes; the portable default | +| charlie12345/ROCmFPX | [`charlie12345/ROCmFPX`](https://github.com/charlie12345/ROCmFPX) (+ reference fork [`rocmfp4-llama`](https://github.com/charlie12345/rocmfp4-llama), clean pkg [`rocmfp4`](https://github.com/charlie12345/rocmfp4)) | Experimental AMD-focused GGUF *weight* quant family layered on llama.cpp | HIP/ROCm + Vulkan (CUDA **disabled**), CPU reference | **Dankclaw** (native Strix Halo) | +| ik_llama.cpp | [`ikawrakow/ik_llama.cpp`](https://github.com/ikawrakow/ik_llama.cpp) | Fork focused on SOTA quants + CPU / hybrid GPU-CPU performance | CPU (AVX-512/VNNI), CUDA (Turing+), Vulkan, ROCm (weak) | **Dankserver** candidate (3090 + MoE) | + +Key lineage notes: +- Mainline moved from `ggerganov/llama.cpp` to the `ggml-org` org. Rolling + commit hashes (the fleet pins `b10121`) plus newer `vX.Y.Z` tags. Pin a + commit for reproducibility. +- ROCmFPX is an *experimental feature family* on the canonical `main` branch; + three GitHub surfaces exist (reference fork, clean patch, pre-patched tree). + It is **AMD-only** — CUDA is compiled out. +- ik_llama.cpp's core is an **August-2024 llama.cpp snapshot** maintained with + a rolling set of PRs (MLA, fused MoE, row-interleaved quants, MTP, DFlash all + appeared here first, later partially merged upstream). The feature surface is + current; the architecture baseline is ~1.5 years behind mainline. + +## Quantization summary + +### Mainline (what the fleet already ships) +- **Base:** FP32 / FP16 / BF16. +- **K-quants** (modern default, two-level block + double-quant): `Q2_K` … + `Q6_K`, with `_S`/`_M`/`_L` precision mixes. `Q4_K_M` is the popular default; + `Q6_K` is "almost lossless." +- **i-quants** (importance-matrix, non-linear, best quality/size at low bits but + slower decode): `IQ4_NL`, `IQ4_XS`, `IQ3_*`, `IQ2_*`. Row-interleaved 3-bit + family (IQ3_XXS/S/XS) reduces the quality penalty of whole-row quant. +- **Legacy:** `Q*_0`/`Q*_1` (fast, simple), ternary `TQ1_0`. +- **KV cache quant:** `f32/f16/bf16/q8_0/q4_0/iq4_nl/q5_0/q5_1`; quantizing KV + requires FlashAttention (`-fa on`). +- Speculative decoding native (MTP, EAGLE3, DFlash, DSpark, `ngram-mod`). + +### ROCmFPX (Dankclaw's AMD weight formats) +- Integer GGUF **weight** formats only this fork reads: **ROCmFP2/3/4/6/8** and + **ROCmI4 (W4A4 IU4 WMMA)**. These are custom `ggml` tensor types + (**`ggml_type` 100–108**) with matching GGUF **`ftype`** (ROCmFP4 family + `100–106`; ROCmFPX family `110–113`; `ROCmI4` = `118`). Stock llama.cpp, + Ollama, and LM Studio **cannot load them**. +- Flagship **ROCmFP4** (`block_rocmfp4`): 32 weights/block, two E4M3 scale + bytes, ~18 bytes/block, ~4.5 bpw, E2M1 nibbles + E4M3 codebook up to 10 + units. Variants: `STRIX` / `STRIX_LEAN`, `FAST`, `COHERENT`, plus `*_AGENT` + / `*_STRIX_MTP` draft-head presets. +- **TurboQuant KV:** `Turbo3_0` (3.5 bpw) / `Turbo4_0` (4.5 bpw) + `TurboQuant V` + (f16 K) on the production model. +- **ROCmI4 (W4A4):** the gfx1151 native IU4 4-bit WMMA path (`-DGGML_HIP_ROCMI4_W4A4=ON`); + lossy activation quant, exact int8 MMQ. + +### ik_llama.cpp (SOTA quants + CPU/hybrid) +- **IQ\*** non-linear family; `IQ4_NL` gives lower perplexity than mainline + `Q4_K_M` at the same size. Improved **K-quants** and the signature + **row-interleaved IQK / R4 / R8** packing (activates AVX-512/VNNI/NEON + matmul paths that vanilla llama.cpp leaves cold). +- **Trellis** quants (`IQ*_KT`), **MXFP4**, **Q8_KV** fast KV type, first-class + **Bitnet**. +- ⚠️ **K-quant / `-rtr` trap (critical for the 3090):** K-quants have **no CUDA + row-interleaved path**; `-rtr` (run-time repack) forces CPU-resident K-quant + tensors to CPU matmul even when GPU would be faster. Prefer IQK/R4/Trellis/ + MXFP4 variants, or skip `-rtr` and let `--fit`/`-ot` place experts. + +## Fit for our servers + +### Dankserver — NVIDIA RTX 3090 (24 GB VRAM), 62 GB RAM, CUDA +Serves **Ornith-1.5-35B-A3B CRACK**, a 35B **MoE** (~36 experts; the CRACK +GGUF README lists 256/8-active — verify `n_expert` from the GGUF). +- **Mainline** is the natural baseline: `Q4_K_M`/`Q5_K_M`, FlashAttention to + shrink KV, `GGML_CUDA_FORCE_MMQ=1` (default on RTX 30xx, lower VRAM), + `--cpu-moe` to spill experts to the 62 GB RAM, and `ngram-mod` drafts (MoEs + want long drafts). +- **ik_llama.cpp** is the more compelling *upgrade* here: fused MoE (`-fmoe`), + **offload-only-active-experts** (`-ooae`) which — with only ~8 active experts — + sharply cuts RAM→VRAM traffic, `--fit`, smart-expert-reduction (`-ser`), GDN + fusion for the SSM path, and MTP. Requires the CUDA-90 + AVX-512 build and care + with the `-rtr` K-quant trap. +- **ROCmFPX** does **not** help a CUDA box (ROCm-only formats, CUDA disabled). + The 3090's lever is CUDA quant + FastMTP, not ROCmFPX. + +### Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified VRAM), ROCm 7.2.4 +Serves **kingjones Qwen3.8-27B ROCmFP4 STRIX MTP** (the production build). +- **ROCmFPX is the native fit**: IU4/MTP/FPX, HIP+Vulkan kernels, TurboQuant KV, + 262K ctx / 1 slot. Measured 8K MTP ~30.3 tok/s (2.83× vs `Q4_K_M`). +- **ik_llama.cpp** — ROCm is its **weakest** backend (gfx1151 not validated) and + the k-quant-no-hip-row-interleave issue bites there too. Keep ROCmFPX. +- **Mainline** ROCm build works for correctness (standard quants) but loses FPX, + IU4, TurboQuant, MTEA, and agent presets. + +## When to reach for which +- **Stay mainline** if you want one portable upstream for both boxes, the widest + GGUF ecosystem, and standard FlashAttention/MTP. Best fit for **Dankserver** + if you don't want the ik_llama.cpp build surface. +- **Use ik_llama.cpp** for Dankserver when the MoE offload and fused-MoE wins + justify an extra build (watch `-rtr` + K-quants). +- **Use ROCmFPX** for Dankclaw — it is purpose-built for gfx1151. +- **ROCmFPX** is for the AMD fleet; **CUDA/FastMTP** for the NVIDIA box. Do not + port ROCmFP4 to the 3090 via Vulkan as a primary path. + +## See Also +- [[Llama Cpp Code Diff]] — the code-level quantization diff + comparison table +- [[dealign-ornith-crack]] — the CRACK model on Dankserver +- [[kingjones-strix-mtp]] — the ROCmFP4 model on Dankclaw +- [[Quantization by Hardware Architecture]] — which quant fits which GPU/CPU +- [[SGLang]] — the structured-generation serving engine (the other lane) + +## Citations +- Mainline — https://github.com/ggml-org/llama.cpp +- charlie12345/ROCmFPX — https://github.com/charlie12345/ROCmFPX · + rocmfp4-llama — https://github.com/charlie12345/rocmfp4-llama · rocmfp4 — + https://github.com/charlie12345/rocmfp4 +- ik_llama.cpp — https://github.com/ikawrakow/ik_llama.cpp · docs · + https://ikawrakow-ik_llama-cpp.mintlify.app/ +- ROCmFPX quant/enum/serve detail — + https://github.com/charlie12345/ROCmFPX/blob/main/ggml/include/ggml.h +- SGLang — https://github.com/sgl-project/sglang · https://docs.sglang.io/ diff --git a/docs/Research/SGLang.md b/docs/Research/SGLang.md new file mode 100644 index 0000000..de6d482 --- /dev/null +++ b/docs/Research/SGLang.md @@ -0,0 +1,154 @@ +--- +type: research +title: SGLang +description: What SGLang is, how it works (RadixAttention, paged KV, overlap scheduler, structured output), how it compares to llama.cpp and vLLM, and whether it fits Dankserver (RTX 3090) and Dankclaw (Strix Halo). +tags: + - sglang + - serving + - inference + - structured-output + - radix-attention + - llama.cpp +timestamp: 2026-09-01 +--- + +# SGLang + +SGLang (for **S**tructured **G**eneration **Lang**uage) is a high-throughput +serving engine for large (and vision-language) models, from the LMSYS / UC +Berkeley team, published at NeurIPS 2024. Today it is one of the two dominant +open-source serving engines, alongside vLLM. People talk about it because it +makes "call the model, branch, force the output into JSON" programs *fast*. + +## What it is — in one paragraph + +You serve an LLM by writing a small program: call the model, then based on its +output call it again, branch into parallel attempts, and force the answer into a +JSON schema. SGLang co-designs a Python-embedded language with a runtime that can +*see* the program's structure, then exploits it: it caches and reuses the model's +"working memory" (the KV cache) across requests that share a prompt prefix, +batches new requests into running generations without waiting for a batch to +drain, and forces tightly-structured output (JSON / regex / tool tags) by masking +illegal tokens one step at a time — or several at once. Reported payoff: +**up to 6.4× higher throughput** on structured / agentic / RAG workloads vs vLLM, +TGI, and similar. + +## Core architecture + +Two halves that can also run independently: +- **Frontend** — a Python DSL (`@sgl.function`, `gen`/`select`/`fork`/`join`/ + `extend`) for multi-call LLM programs with branching, parallel sampling, tool + calls, and structured output. +- **Backend runtime** — a high-throughput server (`python -m sglang.launch_server`) + or an in-process `Engine`. OpenAI-compatible `/v1`, so RadixAttention and + structured decoding work with a vanilla OpenAI client. + +Building blocks: +- **z-normalization** — pre-compile per-request sampling metadata into contiguous + tensors so the GPU sampler does one bulk copy per batch per step (kills the + CPU scheduler bottleneck). +- **PagedTensor** — the KV cache is paged (fixed-size pages) and laid out + non-contiguously in a global pool; mirrors vLLM's PagedAttention. +- **RadixAttention** — a radix tree of shared token prefixes; on a new request it + walks to the longest cached prefix, reuses those KV blocks, and only prefills + the divergent suffix. Payoff is compute + throughput for long shared prefixes + (RAG, agents), not raw memory. +- **Overlap scheduler** — runs the CPU prep for step t+1 *under* the GPU forward + of step t, so kernels launch back-to-back with no idle bubble (plus CUDA + graphs + torch.compile). +- **Structured / output-guided generation** — compiles the constraint (JSON + schema / regex / EBNF) into a finite-state machine over the vocabulary and masks + disallowed logits; **jump-forward** decoding emits forced token runs in one step. + Backends: XGrammar (default), Outlines. + +## SGLang vs llama.cpp vs vLLM + +| Dimension | SGLang | llama.cpp | +|-----------|--------|-----------| +| Runtime | Python + CUDA/C++ kernels | C/C++, no Python required | +| GPU requirement | GPU required, **no CPU path** | CPU-only / GPU-only / hybrid offload | +| Model format | HF safetensors; AWQ/GPTQ/FP8 on the fly; **GGUF via llama.cpp backend** | **GGUF only** (convert first) | +| KV reuse | RadixAttention (always-on radix tree) | basic prompt caching, no radix sharing | +| Structured output | first-class (XGrammar, jump-forward) | narrower | +| Concurrency | continuous batching + overlap scheduler | single-node, lower concurrency | +| Multi-GPU | native tensor parallelism | layer-split, more manual | +| Setup | `pip install sglang[all]`, venv, CUDA match | static binary / Docker, minutes | +| License | Apache 2.0 | MIT | + +**Pick SGLang** when you need many concurrent users, structured / JSON / tool-call +output, or repeated long prefixes (RAG, agents) where RadixAttention pays off, and +you own a GPU + Python/CUDA dependency chain. **Pick llama.cpp** for the fastest +download-to-endpoint path, CPU/edge, GGUF libraries, and quantization breadth. +**Pick vLLM** for the broadest model-hardware coverage and mature multi-GPU. + +The paper's headline 6.4× / 5× numbers are vs **vLLM v0.1** (no prefix caching); +the gap narrows vs modern vLLM — benchmark on your own traffic. + +## Deployment model & gotchas +- **Pluggable backends:** default vLLM engine, SGLang's own `sglang` engine, or a + recent beta **`llama.cpp` backend** that serves GGUF and reuses `llama-server`. +- **VRAM headroom runs higher** than llama.cpp (pre-allocated KV pool + radix + tree). Size with `--mem-fraction-static`, `--context-length`, + `--chunked-prefill-size`. OOM → lower context / fraction or a smaller quant. +- **Model-format lock-in:** wants safetensors + AWQ/GPTQ/FP8 (or GGUF via the + llama.cpp backend). The current CPU-Q6_K_P and ROCmFP4-MTP models are **GGUF**. +- **AMD / ROCm: yes, officially.** Prebuilt `lmsysorg/sglang` ROCm Docker image or + `pip`. Quant on AMD: FP8 (Aiter/Triton), AWQ (Triton), MXFP4 (CDNA3/4), + `petit_nvfp4`. Marlin-based methods are **out** on AMD. Kernel work targets + **datacenter Instinct (CDNA2/3/4)** — consumer/client parts like **Strix Halo + (gfx1151)** are supported at the ROCm level but are not the kernel-tuning target. +- **Prefix diversity kills RadixAttention** (per-user/personalized prompts → tree + degenerates). Single-replica trees need prefix-aware routing across a fleet + (`sglang-router --policy cache_aware`). Fast-moving API — pin versions. + +## Relevance to our servers + +### Dankserver — NVIDIA RTX 3090 (24 GB) + 62 GB RAM — 35B MoE +A **genuine fit candidate (with a migration step)**. The 3090 is CUDA-native. +A 35B MoE (~3B active) is ideal for continuous batching + structured generation, +and 24 GB can hold a quantized 35B MoE (FP8/AWQ safetensors ~18–20 GB) with room +for KV. MoE expert parallelism is supported. +- **What changes:** the current model is a **CPU GGUF (Q6_K_P)**; SGLang wants + safetensors (FP8/AWQ) for CUDA, OR use SGLang's **`llama.cpp` backend** to serve + the existing GGUF under SGLang's frontend. Check for an AWQ/FP8 variant of the + HauhauCS checkpoint on Hugging Face. +- **Where it wins:** repeated structured/JSON/tool calls or a long shared system + prompt (RAG) → RadixAttention + jump-forward give big gains vs the current + sequential CPU pipeline. Simple low-concurrency chat → the operational weight + may not pay off. + +### Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified) — Qwen3.8-27B ROCmFP4 MTP +A **niche / experimental fit, not a drop-in replacement**. 64 GiB is generous and +SGLang-on-ROCm is supported (Docker/pip), but: +- Our model is a **custom ROCmFP4/MTP GGUF**. SGLang's primary path wants + safetensors + an AMD quant (FP8/AWQ); MXFP4 needs CDNA3/4 and Strix Halo (CDNA + 3.5 *client*) is **not** an SGLang/AMD kernel target. Quant availability for this + exact MTP model is unverified, and we'd lose the bespoke MTP draft-head speedup. +- **Best-fit path:** SGLang's **`llama.cpp` backend** on top of our existing ROCm + llama.cpp / GGUF — gain the structured-output DSL, RadixAttention, and scheduler + without rebuilding the kernel stack. Or test the native `lmsysorg/sglang` ROCm + image as a proof-of-concept. Treat it as an enhancement/experiment, not a + production swap, until a quant for the MTP model exists and perf is measured on + gfx1151. + +## Tradeoffs at a glance +- Higher baseline VRAM than llama.cpp (KV pool + radix tree). +- GPU/accelerator required — no pure-CPU path (llama.cpp is the only option of the + trio for a CPU-only box). +- Model-format lock-in (safetensors/AWQ/GPTQ/FP8, or GGUF via llama.cpp backend). +- Marlin-based quant out on AMD; Strix Halo is not an MXFP4 target. +- Prefix diversity can nullify RadixAttention; pin versions; watch cache-report. + +## See Also +- [[Llama Cpp Runtimes]] / [[Llama Cpp Code Diff]] — the llama.cpp side of the trio +- [[dealign-ornith-crack]] — the CRACK MoE on Dankserver +- [[kingjones-strix-mtp]] — the ROCmFP4 MTP model on Dankclaw +- [[Quantization by Hardware Architecture]] + +## Citations +- SGLang paper (arXiv:2312.07104) — https://arxiv.org/abs/2312.07104 · NeurIPS 2024 +- Repo — https://github.com/sgl-project/sglang · docs — https://docs.sglang.io/ +- AMD / ROCm — https://docs.sglang.io/docs/hardware-platforms/amd_gpu · + Structured Outputs — https://docs.sglang.io/docs/advanced_features/structured_outputs +- lmsysorg/sglang ROCm images — https://hub.docker.com/r/lmsysorg/sglang +- SGLang vs llama.cpp — https://markaicode.com/vs/sglang-vs-llamacpp/ diff --git a/docs/Research/index.md b/docs/Research/index.md new file mode 100644 index 0000000..52f84d2 --- /dev/null +++ b/docs/Research/index.md @@ -0,0 +1,24 @@ +# Research + +Subagent research briefs for the two hosts. Each page is a focused research +findings doc; the project root index ([docs/index.md](../index.md)) links back. + +## LLM Inference + +- [[Llama Cpp Runtimes]] — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: positioning, quantization, server fit +- [[Llama Cpp Code Diff]] — code-level quantization diff + comparison table + server recommendations +- [[SGLang]] — the structured-generation serving engine (llama.cpp/vLLM comparison, our- servers fit) + +## Image / Media + +- [[ComfyUI ROCm and Vulkan]] — live-verified GPU path for ComfyUI on Strix Halo +- [[Lemonade and sd.cpp]] — the ggml-family image-generation alternative +- [[Jellyfin]] — media server research brief + +## Backups + +- [[Backblaze Integration]] — off-site backup plan (future) + +## Legacy + +- [[Service Research Brief]] — consolidated subagent research behind these pages diff --git a/docs/index.md b/docs/index.md index 8da7500..504cad6 100644 --- a/docs/index.md +++ b/docs/index.md @@ -88,6 +88,9 @@ covers both workloads). - [[Service Research Brief]] — consolidated subagent research behind these pages - [[ComfyUI ROCm and Vulkan]] — live-verified GPU path for ComfyUI on Strix Halo - [[Lemonade and sd.cpp]] — the ggml-family image-generation alternative +- [[Llama Cpp Runtimes]] — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: quantization + server fit +- [[Llama Cpp Code Diff]] — code-level quantization diff of the three llama.cpp runtimes +- [[SGLang]] — the structured-generation serving engine (llama.cpp/vLLM comparison + our fit) - [[Backblaze Integration]] — off-site backup plan (future) - [[Jellyfin]] — media server research brief - [[ARR-Stack]] — automated download stack research brief @@ -159,3 +162,4 @@ The LLM inference fleet on the two hosts. Full index: - [AGENTS.md](../AGENTS.md) — repository rules - [docs/LLM/index.md](LLM/index.md) — the LLM fleet knowledge base +- [dankware Configuration Knowledgebase](~/.pi/agent/docs/index.md) — the "root" level system knowlegebase diff --git a/scripts/test-changed.sh b/scripts/test-changed.sh index afe3254..062fb96 100755 --- a/scripts/test-changed.sh +++ b/scripts/test-changed.sh @@ -131,6 +131,10 @@ for path in "${CHANGED[@]}"; do dankclaw/hermes/*) add_test tests/docs_completeness.bats ;; + AGENTS.md) + # The authoring rulebook; validate the docs vault whenever it changes. + add_test tests/docs_completeness.bats + ;; *) die "no focused-test mapping for '$path'; add one before committing" ;;