---
type: reference
title: SGLang
description: "What SGLang is, how it schedules and caches, how it compares to the runtimes here, and whether it fits this fleet."
tags:
  - sglang
  - serving
  - inference
  - structured-output
  - radix-attention
  - llama.cpp
timestamp: 2026-09-01
---

# SGLang

SGLang (for **S**tructured **G**eneration **Lang**uage) is a high-throughput
serving engine for large (and vision-language) models, from the LMSYS / UC
Berkeley team, published at NeurIPS 2024. Today it is one of the two dominant
open-source serving engines, alongside vLLM. People talk about it because it
makes "call the model, branch, force the output into JSON" programs *fast*.

## What it is — in one paragraph

You serve an LLM by writing a small program: call the model, then based on its
output call it again, branch into parallel attempts, and force the answer into a
JSON schema. SGLang co-designs a Python-embedded language with a runtime that can
*see* the program's structure, then exploits it: it caches and reuses the model's
"working memory" (the KV cache) across requests that share a prompt prefix,
batches new requests into running generations without waiting for a batch to
drain, and forces tightly-structured output (JSON / regex / tool tags) by masking
illegal tokens one step at a time — or several at once. Reported payoff:
**up to 6.4× higher throughput** on structured / agentic / RAG workloads vs vLLM,
TGI, and similar.

## Core architecture

Two halves that can also run independently:
- **Frontend** — a Python DSL (`@sgl.function`, `gen`/`select`/`fork`/`join`/
  `extend`) for multi-call LLM programs with branching, parallel sampling, tool
  calls, and structured output.
- **Backend runtime** — a high-throughput server (`python -m sglang.launch_server`)
  or an in-process `Engine`. OpenAI-compatible `/v1`, so RadixAttention and
  structured decoding work with a vanilla OpenAI client.

Building blocks:
- **z-normalization** — pre-compile per-request sampling metadata into contiguous
  tensors so the GPU sampler does one bulk copy per batch per step (kills the
  CPU scheduler bottleneck).
- **PagedTensor** — the KV cache is paged (fixed-size pages) and laid out
  non-contiguously in a global pool; mirrors vLLM's PagedAttention.
- **RadixAttention** — a radix tree of shared token prefixes; on a new request it
  walks to the longest cached prefix, reuses those KV blocks, and only prefills
  the divergent suffix. Payoff is compute + throughput for long shared prefixes
  (RAG, agents), not raw memory.
- **Overlap scheduler** — runs the CPU prep for step t+1 *under* the GPU forward
  of step t, so kernels launch back-to-back with no idle bubble (plus CUDA
  graphs + torch.compile).
- **Structured / output-guided generation** — compiles the constraint (JSON
  schema / regex / EBNF) into a finite-state machine over the vocabulary and masks
  disallowed logits; **jump-forward** decoding emits forced token runs in one step.
  Backends: XGrammar (default), Outlines.

## SGLang vs llama.cpp vs vLLM

| Dimension | SGLang | llama.cpp |
|-----------|--------|-----------|
| Runtime | Python + CUDA/C++ kernels | C/C++, no Python required |
| GPU requirement | GPU required, **no CPU path** | CPU-only / GPU-only / hybrid offload |
| Model format | HF safetensors; AWQ/GPTQ/FP8 on the fly; **GGUF via llama.cpp backend** | **GGUF only** (convert first) |
| KV reuse | RadixAttention (always-on radix tree) | basic prompt caching, no radix sharing |
| Structured output | first-class (XGrammar, jump-forward) | narrower |
| Concurrency | continuous batching + overlap scheduler | single-node, lower concurrency |
| Multi-GPU | native tensor parallelism | layer-split, more manual |
| Setup | `pip install sglang[all]`, venv, CUDA match | static binary / Docker, minutes |
| License | Apache 2.0 | MIT |

**Pick SGLang** when you need many concurrent users, structured / JSON / tool-call
output, or repeated long prefixes (RAG, agents) where RadixAttention pays off, and
you own a GPU + Python/CUDA dependency chain. **Pick llama.cpp** for the fastest
download-to-endpoint path, CPU/edge, GGUF libraries, and quantization breadth.
**Pick vLLM** for the broadest model-hardware coverage and mature multi-GPU.

The paper's headline 6.4× / 5× numbers are vs **vLLM v0.1** (no prefix caching);
the gap narrows vs modern vLLM — benchmark on your own traffic.

## Deployment model & gotchas
- **Pluggable backends:** default vLLM engine, SGLang's own `sglang` engine, or a
  recent beta **`llama.cpp` backend** that serves GGUF and reuses `llama-server`.
- **VRAM headroom runs higher** than llama.cpp (pre-allocated KV pool + radix
  tree). Size with `--mem-fraction-static`, `--context-length`,
  `--chunked-prefill-size`. OOM → lower context / fraction or a smaller quant.
- **Model-format lock-in:** wants safetensors + AWQ/GPTQ/FP8 (or GGUF via the
  llama.cpp backend). The current CPU-Q6_K_P and ROCmFP4-MTP models are **GGUF**.
- **AMD / ROCm: yes, officially.** Prebuilt `lmsysorg/sglang` ROCm Docker image or
  `pip`. Quant on AMD: FP8 (Aiter/Triton), AWQ (Triton), MXFP4 (CDNA3/4),
  `petit_nvfp4`. Marlin-based methods are **out** on AMD. Kernel work targets
  **datacenter Instinct (CDNA2/3/4)** — consumer/client parts like **Strix Halo
  (gfx1151)** are supported at the ROCm level but are not the kernel-tuning target.
- **Prefix diversity kills RadixAttention** (per-user/personalized prompts → tree
  degenerates). Single-replica trees need prefix-aware routing across a fleet
  (`sglang-router --policy cache_aware`). Fast-moving API — pin versions.

## Relevance to our servers

### Dankserver — NVIDIA RTX 3090 (24 GB) + 62 GB RAM — 35B MoE
A **genuine fit candidate (with a migration step)**. The 3090 is CUDA-native.
A 35B MoE (~3B active) is ideal for continuous batching + structured generation,
and 24 GB can hold a quantized 35B MoE (FP8/AWQ safetensors ~18–20 GB) with room
for KV. MoE expert parallelism is supported.
- **What changes:** the current model is a **CPU GGUF (Q6_K_P)**; SGLang wants
  safetensors (FP8/AWQ) for CUDA, OR use SGLang's **`llama.cpp` backend** to serve
  the existing GGUF under SGLang's frontend. Check for an AWQ/FP8 variant of the
  HauhauCS checkpoint on Hugging Face.
- **Where it wins:** repeated structured/JSON/tool calls or a long shared system
  prompt (RAG) → RadixAttention + jump-forward give big gains vs the current
  sequential CPU pipeline. Simple low-concurrency chat → the operational weight
  may not pay off.

### Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified) — Qwen3.8-27B ROCmFP4 MTP
A **niche / experimental fit, not a drop-in replacement**. 64 GiB is generous and
SGLang-on-ROCm is supported (Docker/pip), but:
- Our model is a **custom ROCmFP4/MTP GGUF**. SGLang's primary path wants
  safetensors + an AMD quant (FP8/AWQ); MXFP4 needs CDNA3/4 and Strix Halo (CDNA
  3.5 *client*) is **not** an SGLang/AMD kernel target. Quant availability for this
  exact MTP model is unverified, and we'd lose the bespoke MTP draft-head speedup.
- **Best-fit path:** SGLang's **`llama.cpp` backend** on top of our existing ROCm
  llama.cpp / GGUF — gain the structured-output DSL, RadixAttention, and scheduler
  without rebuilding the kernel stack. Or test the native `lmsysorg/sglang` ROCm
  image as a proof-of-concept. Treat it as an enhancement/experiment, not a
  production swap, until a quant for the MTP model exists and perf is measured on
  gfx1151.

## Tradeoffs at a glance
- Higher baseline VRAM than llama.cpp (KV pool + radix tree).
- GPU/accelerator required — no pure-CPU path (llama.cpp is the only option of the
  trio for a CPU-only box).
- Model-format lock-in (safetensors/AWQ/GPTQ/FP8, or GGUF via llama.cpp backend).
- Marlin-based quant out on AMD; Strix Halo is not an MXFP4 target.
- Prefix diversity can nullify RadixAttention; pin versions; watch cache-report.

## See Also
- [[Llama Cpp Runtimes]] / [[Llama Cpp Code Diff]] — the llama.cpp side of the trio
- [[dealign-ornith-crack]] — the CRACK MoE on Dankserver
- [[frank-dragoon]] — the endpoint the Dankclaw ROCmFP4 MTP era became (an Eras row there)
- [[Quantization by Hardware Architecture]]

## Citations
- SGLang paper (arXiv:2312.07104) — https://arxiv.org/abs/2312.07104 · NeurIPS 2024
- Repo — https://github.com/sgl-project/sglang · docs — https://docs.sglang.io/
- AMD / ROCm — https://docs.sglang.io/docs/hardware-platforms/amd_gpu ·
  Structured Outputs — https://docs.sglang.io/docs/advanced_features/structured_outputs
- lmsysorg/sglang ROCm images — https://hub.docker.com/r/lmsysorg/sglang
- SGLang vs llama.cpp — https://markaicode.com/vs/sglang-vs-llamacpp/