serverAppConfigs/serverAppConfigs-docs/Research/SGLang.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	SGLang	What SGLang is, how it schedules and caches, how it compares to the runtimes here, and whether it fits this fleet.	sglang serving inference structured-output radix-attention llama.cpp	2026-09-01

SGLang

SGLang (for Structured Generation Language) is a high-throughput serving engine for large (and vision-language) models, from the LMSYS / UC Berkeley team, published at NeurIPS 2024. Today it is one of the two dominant open-source serving engines, alongside vLLM. People talk about it because it makes "call the model, branch, force the output into JSON" programs fast.

What it is — in one paragraph

You serve an LLM by writing a small program: call the model, then based on its output call it again, branch into parallel attempts, and force the answer into a JSON schema. SGLang co-designs a Python-embedded language with a runtime that can see the program's structure, then exploits it: it caches and reuses the model's "working memory" (the KV cache) across requests that share a prompt prefix, batches new requests into running generations without waiting for a batch to drain, and forces tightly-structured output (JSON / regex / tool tags) by masking illegal tokens one step at a time — or several at once. Reported payoff: up to 6.4× higher throughput on structured / agentic / RAG workloads vs vLLM, TGI, and similar.

Core architecture

Two halves that can also run independently:

• Frontend — a Python DSL (@sgl.function, gen/select/fork/join/ extend) for multi-call LLM programs with branching, parallel sampling, tool calls, and structured output.
• Backend runtime — a high-throughput server (python -m sglang.launch_server) or an in-process Engine. OpenAI-compatible /v1, so RadixAttention and structured decoding work with a vanilla OpenAI client.

Building blocks:

• z-normalization — pre-compile per-request sampling metadata into contiguous tensors so the GPU sampler does one bulk copy per batch per step (kills the CPU scheduler bottleneck).
• PagedTensor — the KV cache is paged (fixed-size pages) and laid out non-contiguously in a global pool; mirrors vLLM's PagedAttention.
• RadixAttention — a radix tree of shared token prefixes; on a new request it walks to the longest cached prefix, reuses those KV blocks, and only prefills the divergent suffix. Payoff is compute + throughput for long shared prefixes (RAG, agents), not raw memory.
• Overlap scheduler — runs the CPU prep for step t+1 under the GPU forward of step t, so kernels launch back-to-back with no idle bubble (plus CUDA graphs + torch.compile).
• Structured / output-guided generation — compiles the constraint (JSON schema / regex / EBNF) into a finite-state machine over the vocabulary and masks disallowed logits; jump-forward decoding emits forced token runs in one step. Backends: XGrammar (default), Outlines.

SGLang vs llama.cpp vs vLLM

Dimension	SGLang	llama.cpp
Runtime	Python + CUDA/C++ kernels	C/C++, no Python required
GPU requirement	GPU required, no CPU path	CPU-only / GPU-only / hybrid offload
Model format	HF safetensors; AWQ/GPTQ/FP8 on the fly; GGUF via llama.cpp backend	GGUF only (convert first)
KV reuse	RadixAttention (always-on radix tree)	basic prompt caching, no radix sharing
Structured output	first-class (XGrammar, jump-forward)	narrower
Concurrency	continuous batching + overlap scheduler	single-node, lower concurrency
Multi-GPU	native tensor parallelism	layer-split, more manual
Setup	pip install sglang[all], venv, CUDA match	static binary / Docker, minutes
License	Apache 2.0	MIT

Pick SGLang when you need many concurrent users, structured / JSON / tool-call output, or repeated long prefixes (RAG, agents) where RadixAttention pays off, and you own a GPU + Python/CUDA dependency chain. Pick llama.cpp for the fastest download-to-endpoint path, CPU/edge, GGUF libraries, and quantization breadth. Pick vLLM for the broadest model-hardware coverage and mature multi-GPU.

The paper's headline 6.4× / 5× numbers are vs vLLM v0.1 (no prefix caching); the gap narrows vs modern vLLM — benchmark on your own traffic.

Deployment model & gotchas

• Pluggable backends: default vLLM engine, SGLang's own sglang engine, or a recent beta llama.cpp backend that serves GGUF and reuses llama-server.
• VRAM headroom runs higher than llama.cpp (pre-allocated KV pool + radix tree). Size with --mem-fraction-static, --context-length, --chunked-prefill-size. OOM → lower context / fraction or a smaller quant.
• Model-format lock-in: wants safetensors + AWQ/GPTQ/FP8 (or GGUF via the llama.cpp backend). The current CPU-Q6_K_P and ROCmFP4-MTP models are GGUF.
• AMD / ROCm: yes, officially. Prebuilt lmsysorg/sglang ROCm Docker image or pip. Quant on AMD: FP8 (Aiter/Triton), AWQ (Triton), MXFP4 (CDNA3/4), petit_nvfp4. Marlin-based methods are out on AMD. Kernel work targets datacenter Instinct (CDNA2/3/4) — consumer/client parts like Strix Halo (gfx1151) are supported at the ROCm level but are not the kernel-tuning target.
• Prefix diversity kills RadixAttention (per-user/personalized prompts → tree degenerates). Single-replica trees need prefix-aware routing across a fleet (sglang-router --policy cache_aware). Fast-moving API — pin versions.

Relevance to our servers

Dankserver — NVIDIA RTX 3090 (24 GB) + 62 GB RAM — 35B MoE

A genuine fit candidate (with a migration step). The 3090 is CUDA-native. A 35B MoE (~3B active) is ideal for continuous batching + structured generation, and 24 GB can hold a quantized 35B MoE (FP8/AWQ safetensors ~18–20 GB) with room for KV. MoE expert parallelism is supported.

• What changes: the current model is a CPU GGUF (Q6_K_P); SGLang wants safetensors (FP8/AWQ) for CUDA, OR use SGLang's llama.cpp backend to serve the existing GGUF under SGLang's frontend. Check for an AWQ/FP8 variant of the HauhauCS checkpoint on Hugging Face.
• Where it wins: repeated structured/JSON/tool calls or a long shared system prompt (RAG) → RadixAttention + jump-forward give big gains vs the current sequential CPU pipeline. Simple low-concurrency chat → the operational weight may not pay off.

Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified) — Qwen3.8-27B ROCmFP4 MTP

A niche / experimental fit, not a drop-in replacement. 64 GiB is generous and SGLang-on-ROCm is supported (Docker/pip), but:

• Our model is a custom ROCmFP4/MTP GGUF. SGLang's primary path wants safetensors + an AMD quant (FP8/AWQ); MXFP4 needs CDNA3/4 and Strix Halo (CDNA 3.5 client) is not an SGLang/AMD kernel target. Quant availability for this exact MTP model is unverified, and we'd lose the bespoke MTP draft-head speedup.
• Best-fit path: SGLang's llama.cpp backend on top of our existing ROCm llama.cpp / GGUF — gain the structured-output DSL, RadixAttention, and scheduler without rebuilding the kernel stack. Or test the native lmsysorg/sglang ROCm image as a proof-of-concept. Treat it as an enhancement/experiment, not a production swap, until a quant for the MTP model exists and perf is measured on gfx1151.

Tradeoffs at a glance

• Higher baseline VRAM than llama.cpp (KV pool + radix tree).
• GPU/accelerator required — no pure-CPU path (llama.cpp is the only option of the trio for a CPU-only box).
• Model-format lock-in (safetensors/AWQ/GPTQ/FP8, or GGUF via llama.cpp backend).
• Marlin-based quant out on AMD; Strix Halo is not an MXFP4 target.
• Prefix diversity can nullify RadixAttention; pin versions; watch cache-report.

See Also

• Llama Cpp Runtimes / Llama Cpp Code Diff — the llama.cpp side of the trio
• dealign-ornith-crack — the CRACK MoE on Dankserver
• frank-dragoon — the endpoint the Dankclaw ROCmFP4 MTP era became (an Eras row there)
• Quantization by Hardware Architecture

Citations

• SGLang paper (arXiv:2312.07104) — https://arxiv.org/abs/2312.07104 · NeurIPS 2024
• Repo — https://github.com/sgl-project/sglang · docs — https://docs.sglang.io/
• AMD / ROCm — https://docs.sglang.io/docs/hardware-platforms/amd_gpu · Structured Outputs — https://docs.sglang.io/docs/advanced_features/structured_outputs
• lmsysorg/sglang ROCm images — https://hub.docker.com/r/lmsysorg/sglang
• SGLang vs llama.cpp — https://markaicode.com/vs/sglang-vs-llamacpp/