Frank
929796352a
Fan out subagents to compare mainline llama.cpp, charlie12345/ROCmFPX, and ik_llama.cpp, then add their findings to the project knowledge base: - docs/Research/Llama Cpp Runtimes.md (positioning + quantization + server fit) - docs/Research/Llama Cpp Code Diff.md (code-level quant diff, table, recommendations) - docs/Research/SGLang.md (what SGLang is, comparison, our-boxes fit) - docs/Research/index.md (Research progressive-disclosure index) Cross-linked from docs/index.md (root) and docs/LLM/index.md. AGENTS.md: - Add Documentation Authoring section: docs/ is an OKF bundle; point doc authors at the canonical spec in the system-config knowledge base (~/.pi/agent/docs/references/okf-spec.md), not the orphaned docs/references/okf-spec.md; consolidate read-index/one-concept-per-file/ progressive-disclosure/citation rules. - Map AGENTS.md to docs_completeness.bats in the focused pre-commit hook (it was last committed before the hook existed and had no mapping).
155 lines
8.6 KiB
Markdown
155 lines
8.6 KiB
Markdown
---
|
||
type: research
|
||
title: SGLang
|
||
description: What SGLang is, how it works (RadixAttention, paged KV, overlap scheduler, structured output), how it compares to llama.cpp and vLLM, and whether it fits Dankserver (RTX 3090) and Dankclaw (Strix Halo).
|
||
tags:
|
||
- sglang
|
||
- serving
|
||
- inference
|
||
- structured-output
|
||
- radix-attention
|
||
- llama.cpp
|
||
timestamp: 2026-09-01
|
||
---
|
||
|
||
# SGLang
|
||
|
||
SGLang (for **S**tructured **G**eneration **Lang**uage) is a high-throughput
|
||
serving engine for large (and vision-language) models, from the LMSYS / UC
|
||
Berkeley team, published at NeurIPS 2024. Today it is one of the two dominant
|
||
open-source serving engines, alongside vLLM. People talk about it because it
|
||
makes "call the model, branch, force the output into JSON" programs *fast*.
|
||
|
||
## What it is — in one paragraph
|
||
|
||
You serve an LLM by writing a small program: call the model, then based on its
|
||
output call it again, branch into parallel attempts, and force the answer into a
|
||
JSON schema. SGLang co-designs a Python-embedded language with a runtime that can
|
||
*see* the program's structure, then exploits it: it caches and reuses the model's
|
||
"working memory" (the KV cache) across requests that share a prompt prefix,
|
||
batches new requests into running generations without waiting for a batch to
|
||
drain, and forces tightly-structured output (JSON / regex / tool tags) by masking
|
||
illegal tokens one step at a time — or several at once. Reported payoff:
|
||
**up to 6.4× higher throughput** on structured / agentic / RAG workloads vs vLLM,
|
||
TGI, and similar.
|
||
|
||
## Core architecture
|
||
|
||
Two halves that can also run independently:
|
||
- **Frontend** — a Python DSL (`@sgl.function`, `gen`/`select`/`fork`/`join`/
|
||
`extend`) for multi-call LLM programs with branching, parallel sampling, tool
|
||
calls, and structured output.
|
||
- **Backend runtime** — a high-throughput server (`python -m sglang.launch_server`)
|
||
or an in-process `Engine`. OpenAI-compatible `/v1`, so RadixAttention and
|
||
structured decoding work with a vanilla OpenAI client.
|
||
|
||
Building blocks:
|
||
- **z-normalization** — pre-compile per-request sampling metadata into contiguous
|
||
tensors so the GPU sampler does one bulk copy per batch per step (kills the
|
||
CPU scheduler bottleneck).
|
||
- **PagedTensor** — the KV cache is paged (fixed-size pages) and laid out
|
||
non-contiguously in a global pool; mirrors vLLM's PagedAttention.
|
||
- **RadixAttention** — a radix tree of shared token prefixes; on a new request it
|
||
walks to the longest cached prefix, reuses those KV blocks, and only prefills
|
||
the divergent suffix. Payoff is compute + throughput for long shared prefixes
|
||
(RAG, agents), not raw memory.
|
||
- **Overlap scheduler** — runs the CPU prep for step t+1 *under* the GPU forward
|
||
of step t, so kernels launch back-to-back with no idle bubble (plus CUDA
|
||
graphs + torch.compile).
|
||
- **Structured / output-guided generation** — compiles the constraint (JSON
|
||
schema / regex / EBNF) into a finite-state machine over the vocabulary and masks
|
||
disallowed logits; **jump-forward** decoding emits forced token runs in one step.
|
||
Backends: XGrammar (default), Outlines.
|
||
|
||
## SGLang vs llama.cpp vs vLLM
|
||
|
||
| Dimension | SGLang | llama.cpp |
|
||
|-----------|--------|-----------|
|
||
| Runtime | Python + CUDA/C++ kernels | C/C++, no Python required |
|
||
| GPU requirement | GPU required, **no CPU path** | CPU-only / GPU-only / hybrid offload |
|
||
| Model format | HF safetensors; AWQ/GPTQ/FP8 on the fly; **GGUF via llama.cpp backend** | **GGUF only** (convert first) |
|
||
| KV reuse | RadixAttention (always-on radix tree) | basic prompt caching, no radix sharing |
|
||
| Structured output | first-class (XGrammar, jump-forward) | narrower |
|
||
| Concurrency | continuous batching + overlap scheduler | single-node, lower concurrency |
|
||
| Multi-GPU | native tensor parallelism | layer-split, more manual |
|
||
| Setup | `pip install sglang[all]`, venv, CUDA match | static binary / Docker, minutes |
|
||
| License | Apache 2.0 | MIT |
|
||
|
||
**Pick SGLang** when you need many concurrent users, structured / JSON / tool-call
|
||
output, or repeated long prefixes (RAG, agents) where RadixAttention pays off, and
|
||
you own a GPU + Python/CUDA dependency chain. **Pick llama.cpp** for the fastest
|
||
download-to-endpoint path, CPU/edge, GGUF libraries, and quantization breadth.
|
||
**Pick vLLM** for the broadest model-hardware coverage and mature multi-GPU.
|
||
|
||
The paper's headline 6.4× / 5× numbers are vs **vLLM v0.1** (no prefix caching);
|
||
the gap narrows vs modern vLLM — benchmark on your own traffic.
|
||
|
||
## Deployment model & gotchas
|
||
- **Pluggable backends:** default vLLM engine, SGLang's own `sglang` engine, or a
|
||
recent beta **`llama.cpp` backend** that serves GGUF and reuses `llama-server`.
|
||
- **VRAM headroom runs higher** than llama.cpp (pre-allocated KV pool + radix
|
||
tree). Size with `--mem-fraction-static`, `--context-length`,
|
||
`--chunked-prefill-size`. OOM → lower context / fraction or a smaller quant.
|
||
- **Model-format lock-in:** wants safetensors + AWQ/GPTQ/FP8 (or GGUF via the
|
||
llama.cpp backend). The current CPU-Q6_K_P and ROCmFP4-MTP models are **GGUF**.
|
||
- **AMD / ROCm: yes, officially.** Prebuilt `lmsysorg/sglang` ROCm Docker image or
|
||
`pip`. Quant on AMD: FP8 (Aiter/Triton), AWQ (Triton), MXFP4 (CDNA3/4),
|
||
`petit_nvfp4`. Marlin-based methods are **out** on AMD. Kernel work targets
|
||
**datacenter Instinct (CDNA2/3/4)** — consumer/client parts like **Strix Halo
|
||
(gfx1151)** are supported at the ROCm level but are not the kernel-tuning target.
|
||
- **Prefix diversity kills RadixAttention** (per-user/personalized prompts → tree
|
||
degenerates). Single-replica trees need prefix-aware routing across a fleet
|
||
(`sglang-router --policy cache_aware`). Fast-moving API — pin versions.
|
||
|
||
## Relevance to our servers
|
||
|
||
### Dankserver — NVIDIA RTX 3090 (24 GB) + 62 GB RAM — 35B MoE
|
||
A **genuine fit candidate (with a migration step)**. The 3090 is CUDA-native.
|
||
A 35B MoE (~3B active) is ideal for continuous batching + structured generation,
|
||
and 24 GB can hold a quantized 35B MoE (FP8/AWQ safetensors ~18–20 GB) with room
|
||
for KV. MoE expert parallelism is supported.
|
||
- **What changes:** the current model is a **CPU GGUF (Q6_K_P)**; SGLang wants
|
||
safetensors (FP8/AWQ) for CUDA, OR use SGLang's **`llama.cpp` backend** to serve
|
||
the existing GGUF under SGLang's frontend. Check for an AWQ/FP8 variant of the
|
||
HauhauCS checkpoint on Hugging Face.
|
||
- **Where it wins:** repeated structured/JSON/tool calls or a long shared system
|
||
prompt (RAG) → RadixAttention + jump-forward give big gains vs the current
|
||
sequential CPU pipeline. Simple low-concurrency chat → the operational weight
|
||
may not pay off.
|
||
|
||
### Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified) — Qwen3.8-27B ROCmFP4 MTP
|
||
A **niche / experimental fit, not a drop-in replacement**. 64 GiB is generous and
|
||
SGLang-on-ROCm is supported (Docker/pip), but:
|
||
- Our model is a **custom ROCmFP4/MTP GGUF**. SGLang's primary path wants
|
||
safetensors + an AMD quant (FP8/AWQ); MXFP4 needs CDNA3/4 and Strix Halo (CDNA
|
||
3.5 *client*) is **not** an SGLang/AMD kernel target. Quant availability for this
|
||
exact MTP model is unverified, and we'd lose the bespoke MTP draft-head speedup.
|
||
- **Best-fit path:** SGLang's **`llama.cpp` backend** on top of our existing ROCm
|
||
llama.cpp / GGUF — gain the structured-output DSL, RadixAttention, and scheduler
|
||
without rebuilding the kernel stack. Or test the native `lmsysorg/sglang` ROCm
|
||
image as a proof-of-concept. Treat it as an enhancement/experiment, not a
|
||
production swap, until a quant for the MTP model exists and perf is measured on
|
||
gfx1151.
|
||
|
||
## Tradeoffs at a glance
|
||
- Higher baseline VRAM than llama.cpp (KV pool + radix tree).
|
||
- GPU/accelerator required — no pure-CPU path (llama.cpp is the only option of the
|
||
trio for a CPU-only box).
|
||
- Model-format lock-in (safetensors/AWQ/GPTQ/FP8, or GGUF via llama.cpp backend).
|
||
- Marlin-based quant out on AMD; Strix Halo is not an MXFP4 target.
|
||
- Prefix diversity can nullify RadixAttention; pin versions; watch cache-report.
|
||
|
||
## See Also
|
||
- [[Llama Cpp Runtimes]] / [[Llama Cpp Code Diff]] — the llama.cpp side of the trio
|
||
- [[dealign-ornith-crack]] — the CRACK MoE on Dankserver
|
||
- [[kingjones-strix-mtp]] — the ROCmFP4 MTP model on Dankclaw
|
||
- [[Quantization by Hardware Architecture]]
|
||
|
||
## Citations
|
||
- SGLang paper (arXiv:2312.07104) — https://arxiv.org/abs/2312.07104 · NeurIPS 2024
|
||
- Repo — https://github.com/sgl-project/sglang · docs — https://docs.sglang.io/
|
||
- AMD / ROCm — https://docs.sglang.io/docs/hardware-platforms/amd_gpu ·
|
||
Structured Outputs — https://docs.sglang.io/docs/advanced_features/structured_outputs
|
||
- lmsysorg/sglang ROCm images — https://hub.docker.com/r/lmsysorg/sglang
|
||
- SGLang vs llama.cpp — https://markaicode.com/vs/sglang-vs-llamacpp/
|