---
type: research
title: SGLang
description: What SGLang is, how it works (RadixAttention, paged KV, overlap scheduler, structured output), how it compares to llama.cpp and vLLM, and whether it fits Dankserver (RTX 3090) and Dankclaw (Strix Halo).
tags:
  - sglang
  - serving
  - inference
  - structured-output
  - radix-attention
  - llama.cpp
timestamp: 2026-09-01
---

# SGLang

SGLang (for **S**tructured **G**eneration **Lang**uage) is a high-throughput
serving engine for large (and vision-language) models, from the LMSYS / UC
Berkeley team, published at NeurIPS 2024. Today it is one of the two dominant
open-source serving engines, alongside vLLM. People talk about it because it
makes "call the model, branch, force the output into JSON" programs *fast*.

## What it is — in one paragraph

You serve an LLM by writing a small program: call the model, then based on its
output call it again, branch into parallel attempts, and force the answer into a
JSON schema. SGLang co-designs a Python-embedded language with a runtime that can
*see* the program's structure, then exploits it: it caches and reuses the model's
"working memory" (the KV cache) across requests that share a prompt prefix,
batches new requests into running generations without waiting for a batch to
drain, and forces tightly-structured output (JSON / regex / tool tags) by masking
illegal tokens one step at a time — or several at once. Reported payoff:
**up to 6.4× higher throughput** on structured / agentic / RAG workloads vs vLLM,
TGI, and similar.

## Core architecture

Two halves that can also run independently:
- **Frontend** — a Python DSL (`@sgl.function`, `gen`/`select`/`fork`/`join`/
  `extend`) for multi-call LLM programs with branching, parallel sampling, tool
  calls, and structured output.
- **Backend runtime** — a high-throughput server (`python -m sglang.launch_server`)
  or an in-process `Engine`. OpenAI-compatible `/v1`, so RadixAttention and
  structured decoding work with a vanilla OpenAI client.

Building blocks:
- **z-normalization** — pre-compile per-request sampling metadata into contiguous
  tensors so the GPU sampler does one bulk copy per batch per step (kills the
  CPU scheduler bottleneck).
- **PagedTensor** — the KV cache is paged (fixed-size pages) and laid out
  non-contiguously in a global pool; mirrors vLLM's PagedAttention.
- **RadixAttention** — a radix tree of shared token prefixes; on a new request it
  walks to the longest cached prefix, reuses those KV blocks, and only prefills
  the divergent suffix. Payoff is compute + throughput for long shared prefixes
  (RAG, agents), not raw memory.
- **Overlap scheduler** — runs the CPU prep for step t+1 *under* the GPU forward
  of step t, so kernels launch back-to-back with no idle bubble (plus CUDA
  graphs + torch.compile).
- **Structured / output-guided generation** — compiles the constraint (JSON
  schema / regex / EBNF) into a finite-state machine over the vocabulary and masks
  disallowed logits; **jump-forward** decoding emits forced token runs in one step.
  Backends: XGrammar (default), Outlines.

## SGLang vs llama.cpp vs vLLM

| Dimension | SGLang | llama.cpp |
|-----------|--------|-----------|
| Runtime | Python + CUDA/C++ kernels | C/C++, no Python required |
| GPU requirement | GPU required, **no CPU path** | CPU-only / GPU-only / hybrid offload |
| Model format | HF safetensors; AWQ/GPTQ/FP8 on the fly; **GGUF via llama.cpp backend** | **GGUF only** (convert first) |
| KV reuse | RadixAttention (always-on radix tree) | basic prompt caching, no radix sharing |
| Structured output | first-class (XGrammar, jump-forward) | narrower |
| Concurrency | continuous batching + overlap scheduler | single-node, lower concurrency |
| Multi-GPU | native tensor parallelism | layer-split, more manual |
| Setup | `pip install sglang[all]`, venv, CUDA match | static binary / Docker, minutes |
| License | Apache 2.0 | MIT |

**Pick SGLang** when you need many concurrent users, structured / JSON / tool-call
output, or repeated long prefixes (RAG, agents) where RadixAttention pays off, and
you own a GPU + Python/CUDA dependency chain. **Pick llama.cpp** for the fastest
download-to-endpoint path, CPU/edge, GGUF libraries, and quantization breadth.
**Pick vLLM** for the broadest model-hardware coverage and mature multi-GPU.

The paper's headline 6.4× / 5× numbers are vs **vLLM v0.1** (no prefix caching);
the gap narrows vs modern vLLM — benchmark on your own traffic.

## Deployment model & gotchas
- **Pluggable backends:** default vLLM engine, SGLang's own `sglang` engine, or a
  recent beta **`llama.cpp` backend** that serves GGUF and reuses `llama-server`.
- **VRAM headroom runs higher** than llama.cpp (pre-allocated KV pool + radix
  tree). Size with `--mem-fraction-static`, `--context-length`,
  `--chunked-prefill-size`. OOM → lower context / fraction or a smaller quant.
- **Model-format lock-in:** wants safetensors + AWQ/GPTQ/FP8 (or GGUF via the
  llama.cpp backend). The current CPU-Q6_K_P and ROCmFP4-MTP models are **GGUF**.
- **AMD / ROCm: yes, officially.** Prebuilt `lmsysorg/sglang` ROCm Docker image or
  `pip`. Quant on AMD: FP8 (Aiter/Triton), AWQ (Triton), MXFP4 (CDNA3/4),
  `petit_nvfp4`. Marlin-based methods are **out** on AMD. Kernel work targets
  **datacenter Instinct (CDNA2/3/4)** — consumer/client parts like **Strix Halo
  (gfx1151)** are supported at the ROCm level but are not the kernel-tuning target.
- **Prefix diversity kills RadixAttention** (per-user/personalized prompts → tree
  degenerates). Single-replica trees need prefix-aware routing across a fleet
  (`sglang-router --policy cache_aware`). Fast-moving API — pin versions.

## Relevance to our servers

### Dankserver — NVIDIA RTX 3090 (24 GB) + 62 GB RAM — 35B MoE
A **genuine fit candidate (with a migration step)**. The 3090 is CUDA-native.
A 35B MoE (~3B active) is ideal for continuous batching + structured generation,
and 24 GB can hold a quantized 35B MoE (FP8/AWQ safetensors ~18–20 GB) with room
for KV. MoE expert parallelism is supported.
- **What changes:** the current model is a **CPU GGUF (Q6_K_P)**; SGLang wants
  safetensors (FP8/AWQ) for CUDA, OR use SGLang's **`llama.cpp` backend** to serve
  the existing GGUF under SGLang's frontend. Check for an AWQ/FP8 variant of the
  HauhauCS checkpoint on Hugging Face.
- **Where it wins:** repeated structured/JSON/tool calls or a long shared system
  prompt (RAG) → RadixAttention + jump-forward give big gains vs the current
  sequential CPU pipeline. Simple low-concurrency chat → the operational weight
  may not pay off.

### Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified) — Qwen3.8-27B ROCmFP4 MTP
A **niche / experimental fit, not a drop-in replacement**. 64 GiB is generous and
SGLang-on-ROCm is supported (Docker/pip), but:
- Our model is a **custom ROCmFP4/MTP GGUF**. SGLang's primary path wants
  safetensors + an AMD quant (FP8/AWQ); MXFP4 needs CDNA3/4 and Strix Halo (CDNA
  3.5 *client*) is **not** an SGLang/AMD kernel target. Quant availability for this
  exact MTP model is unverified, and we'd lose the bespoke MTP draft-head speedup.
- **Best-fit path:** SGLang's **`llama.cpp` backend** on top of our existing ROCm
  llama.cpp / GGUF — gain the structured-output DSL, RadixAttention, and scheduler
  without rebuilding the kernel stack. Or test the native `lmsysorg/sglang` ROCm
  image as a proof-of-concept. Treat it as an enhancement/experiment, not a
  production swap, until a quant for the MTP model exists and perf is measured on
  gfx1151.

## Tradeoffs at a glance
- Higher baseline VRAM than llama.cpp (KV pool + radix tree).
- GPU/accelerator required — no pure-CPU path (llama.cpp is the only option of the
  trio for a CPU-only box).
- Model-format lock-in (safetensors/AWQ/GPTQ/FP8, or GGUF via llama.cpp backend).
- Marlin-based quant out on AMD; Strix Halo is not an MXFP4 target.
- Prefix diversity can nullify RadixAttention; pin versions; watch cache-report.

## See Also
- [[Llama Cpp Runtimes]] / [[Llama Cpp Code Diff]] — the llama.cpp side of the trio
- [[dealign-ornith-crack]] — the CRACK MoE on Dankserver
- [[frank-dragoon]] — the endpoint the Dankclaw ROCmFP4 MTP era became (an Eras row there)
- [[Quantization by Hardware Architecture]]

## Citations
- SGLang paper (arXiv:2312.07104) — https://arxiv.org/abs/2312.07104 · NeurIPS 2024
- Repo — https://github.com/sgl-project/sglang · docs — https://docs.sglang.io/
- AMD / ROCm — https://docs.sglang.io/docs/hardware-platforms/amd_gpu ·
  Structured Outputs — https://docs.sglang.io/docs/advanced_features/structured_outputs
- lmsysorg/sglang ROCm images — https://hub.docker.com/r/lmsysorg/sglang
- SGLang vs llama.cpp — https://markaicode.com/vs/sglang-vs-llamacpp/