Frank
219a097b39
Every payload swap a live endpoint has ever run is now one row in that endpoint's Eras table, with its rollback path in the same row. No per-attempt page, no per-swap ledger, nowhere else. - Added: LLM/frank-archon.md (radiance, 2x R9700 TP=2), LLM/frank-dragoon.md (halogen flash-server, 19 eras), LLM/frank-zealot.md (llama.cpp PrismML fork on the 3090). Each carries Current shape, Eras, Laws, wire contract, files. - Deleted: 45 per-payload runbooks -- the frank-dragoon line (11 including the pre-name halogen/kingjones/kairic/CIRU/eemin/agentionai/dflash2 pages and the benchmark page), the frank-zealot line (llama.cpp/vLLM/ik_llama/SGLang eras), the frank-archon line (vLLM/FreeToken/R9V/llama.cpp/radiance eras), plus frank-medium, frank-npu and the ubatch/PCIe one-offs that owned nothing. - Rewired: every wikilink and markdown-path reference to a folded page points at the endpoint page instead; the three Research see-also lists and the ubatch sweep's runbook link included. No anchored links existed in the set. - Regenerated both index pages as pure locators -- one row, one clause, no numbers -- so the docs_completeness locator law holds for all 153 pages. tests/docs_completeness.bats green: index links every page, no unresolved links. The pages moved to .AGENTTRASH/20261018-doc-dedupe/LLM/ (gitignored, and git history holds every byte).
155 lines
8.6 KiB
Markdown
155 lines
8.6 KiB
Markdown
---
|
||
type: research
|
||
title: SGLang
|
||
description: What SGLang is, how it works (RadixAttention, paged KV, overlap scheduler, structured output), how it compares to llama.cpp and vLLM, and whether it fits Dankserver (RTX 3090) and Dankclaw (Strix Halo).
|
||
tags:
|
||
- sglang
|
||
- serving
|
||
- inference
|
||
- structured-output
|
||
- radix-attention
|
||
- llama.cpp
|
||
timestamp: 2026-09-01
|
||
---
|
||
|
||
# SGLang
|
||
|
||
SGLang (for **S**tructured **G**eneration **Lang**uage) is a high-throughput
|
||
serving engine for large (and vision-language) models, from the LMSYS / UC
|
||
Berkeley team, published at NeurIPS 2024. Today it is one of the two dominant
|
||
open-source serving engines, alongside vLLM. People talk about it because it
|
||
makes "call the model, branch, force the output into JSON" programs *fast*.
|
||
|
||
## What it is — in one paragraph
|
||
|
||
You serve an LLM by writing a small program: call the model, then based on its
|
||
output call it again, branch into parallel attempts, and force the answer into a
|
||
JSON schema. SGLang co-designs a Python-embedded language with a runtime that can
|
||
*see* the program's structure, then exploits it: it caches and reuses the model's
|
||
"working memory" (the KV cache) across requests that share a prompt prefix,
|
||
batches new requests into running generations without waiting for a batch to
|
||
drain, and forces tightly-structured output (JSON / regex / tool tags) by masking
|
||
illegal tokens one step at a time — or several at once. Reported payoff:
|
||
**up to 6.4× higher throughput** on structured / agentic / RAG workloads vs vLLM,
|
||
TGI, and similar.
|
||
|
||
## Core architecture
|
||
|
||
Two halves that can also run independently:
|
||
- **Frontend** — a Python DSL (`@sgl.function`, `gen`/`select`/`fork`/`join`/
|
||
`extend`) for multi-call LLM programs with branching, parallel sampling, tool
|
||
calls, and structured output.
|
||
- **Backend runtime** — a high-throughput server (`python -m sglang.launch_server`)
|
||
or an in-process `Engine`. OpenAI-compatible `/v1`, so RadixAttention and
|
||
structured decoding work with a vanilla OpenAI client.
|
||
|
||
Building blocks:
|
||
- **z-normalization** — pre-compile per-request sampling metadata into contiguous
|
||
tensors so the GPU sampler does one bulk copy per batch per step (kills the
|
||
CPU scheduler bottleneck).
|
||
- **PagedTensor** — the KV cache is paged (fixed-size pages) and laid out
|
||
non-contiguously in a global pool; mirrors vLLM's PagedAttention.
|
||
- **RadixAttention** — a radix tree of shared token prefixes; on a new request it
|
||
walks to the longest cached prefix, reuses those KV blocks, and only prefills
|
||
the divergent suffix. Payoff is compute + throughput for long shared prefixes
|
||
(RAG, agents), not raw memory.
|
||
- **Overlap scheduler** — runs the CPU prep for step t+1 *under* the GPU forward
|
||
of step t, so kernels launch back-to-back with no idle bubble (plus CUDA
|
||
graphs + torch.compile).
|
||
- **Structured / output-guided generation** — compiles the constraint (JSON
|
||
schema / regex / EBNF) into a finite-state machine over the vocabulary and masks
|
||
disallowed logits; **jump-forward** decoding emits forced token runs in one step.
|
||
Backends: XGrammar (default), Outlines.
|
||
|
||
## SGLang vs llama.cpp vs vLLM
|
||
|
||
| Dimension | SGLang | llama.cpp |
|
||
|-----------|--------|-----------|
|
||
| Runtime | Python + CUDA/C++ kernels | C/C++, no Python required |
|
||
| GPU requirement | GPU required, **no CPU path** | CPU-only / GPU-only / hybrid offload |
|
||
| Model format | HF safetensors; AWQ/GPTQ/FP8 on the fly; **GGUF via llama.cpp backend** | **GGUF only** (convert first) |
|
||
| KV reuse | RadixAttention (always-on radix tree) | basic prompt caching, no radix sharing |
|
||
| Structured output | first-class (XGrammar, jump-forward) | narrower |
|
||
| Concurrency | continuous batching + overlap scheduler | single-node, lower concurrency |
|
||
| Multi-GPU | native tensor parallelism | layer-split, more manual |
|
||
| Setup | `pip install sglang[all]`, venv, CUDA match | static binary / Docker, minutes |
|
||
| License | Apache 2.0 | MIT |
|
||
|
||
**Pick SGLang** when you need many concurrent users, structured / JSON / tool-call
|
||
output, or repeated long prefixes (RAG, agents) where RadixAttention pays off, and
|
||
you own a GPU + Python/CUDA dependency chain. **Pick llama.cpp** for the fastest
|
||
download-to-endpoint path, CPU/edge, GGUF libraries, and quantization breadth.
|
||
**Pick vLLM** for the broadest model-hardware coverage and mature multi-GPU.
|
||
|
||
The paper's headline 6.4× / 5× numbers are vs **vLLM v0.1** (no prefix caching);
|
||
the gap narrows vs modern vLLM — benchmark on your own traffic.
|
||
|
||
## Deployment model & gotchas
|
||
- **Pluggable backends:** default vLLM engine, SGLang's own `sglang` engine, or a
|
||
recent beta **`llama.cpp` backend** that serves GGUF and reuses `llama-server`.
|
||
- **VRAM headroom runs higher** than llama.cpp (pre-allocated KV pool + radix
|
||
tree). Size with `--mem-fraction-static`, `--context-length`,
|
||
`--chunked-prefill-size`. OOM → lower context / fraction or a smaller quant.
|
||
- **Model-format lock-in:** wants safetensors + AWQ/GPTQ/FP8 (or GGUF via the
|
||
llama.cpp backend). The current CPU-Q6_K_P and ROCmFP4-MTP models are **GGUF**.
|
||
- **AMD / ROCm: yes, officially.** Prebuilt `lmsysorg/sglang` ROCm Docker image or
|
||
`pip`. Quant on AMD: FP8 (Aiter/Triton), AWQ (Triton), MXFP4 (CDNA3/4),
|
||
`petit_nvfp4`. Marlin-based methods are **out** on AMD. Kernel work targets
|
||
**datacenter Instinct (CDNA2/3/4)** — consumer/client parts like **Strix Halo
|
||
(gfx1151)** are supported at the ROCm level but are not the kernel-tuning target.
|
||
- **Prefix diversity kills RadixAttention** (per-user/personalized prompts → tree
|
||
degenerates). Single-replica trees need prefix-aware routing across a fleet
|
||
(`sglang-router --policy cache_aware`). Fast-moving API — pin versions.
|
||
|
||
## Relevance to our servers
|
||
|
||
### Dankserver — NVIDIA RTX 3090 (24 GB) + 62 GB RAM — 35B MoE
|
||
A **genuine fit candidate (with a migration step)**. The 3090 is CUDA-native.
|
||
A 35B MoE (~3B active) is ideal for continuous batching + structured generation,
|
||
and 24 GB can hold a quantized 35B MoE (FP8/AWQ safetensors ~18–20 GB) with room
|
||
for KV. MoE expert parallelism is supported.
|
||
- **What changes:** the current model is a **CPU GGUF (Q6_K_P)**; SGLang wants
|
||
safetensors (FP8/AWQ) for CUDA, OR use SGLang's **`llama.cpp` backend** to serve
|
||
the existing GGUF under SGLang's frontend. Check for an AWQ/FP8 variant of the
|
||
HauhauCS checkpoint on Hugging Face.
|
||
- **Where it wins:** repeated structured/JSON/tool calls or a long shared system
|
||
prompt (RAG) → RadixAttention + jump-forward give big gains vs the current
|
||
sequential CPU pipeline. Simple low-concurrency chat → the operational weight
|
||
may not pay off.
|
||
|
||
### Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified) — Qwen3.8-27B ROCmFP4 MTP
|
||
A **niche / experimental fit, not a drop-in replacement**. 64 GiB is generous and
|
||
SGLang-on-ROCm is supported (Docker/pip), but:
|
||
- Our model is a **custom ROCmFP4/MTP GGUF**. SGLang's primary path wants
|
||
safetensors + an AMD quant (FP8/AWQ); MXFP4 needs CDNA3/4 and Strix Halo (CDNA
|
||
3.5 *client*) is **not** an SGLang/AMD kernel target. Quant availability for this
|
||
exact MTP model is unverified, and we'd lose the bespoke MTP draft-head speedup.
|
||
- **Best-fit path:** SGLang's **`llama.cpp` backend** on top of our existing ROCm
|
||
llama.cpp / GGUF — gain the structured-output DSL, RadixAttention, and scheduler
|
||
without rebuilding the kernel stack. Or test the native `lmsysorg/sglang` ROCm
|
||
image as a proof-of-concept. Treat it as an enhancement/experiment, not a
|
||
production swap, until a quant for the MTP model exists and perf is measured on
|
||
gfx1151.
|
||
|
||
## Tradeoffs at a glance
|
||
- Higher baseline VRAM than llama.cpp (KV pool + radix tree).
|
||
- GPU/accelerator required — no pure-CPU path (llama.cpp is the only option of the
|
||
trio for a CPU-only box).
|
||
- Model-format lock-in (safetensors/AWQ/GPTQ/FP8, or GGUF via llama.cpp backend).
|
||
- Marlin-based quant out on AMD; Strix Halo is not an MXFP4 target.
|
||
- Prefix diversity can nullify RadixAttention; pin versions; watch cache-report.
|
||
|
||
## See Also
|
||
- [[Llama Cpp Runtimes]] / [[Llama Cpp Code Diff]] — the llama.cpp side of the trio
|
||
- [[dealign-ornith-crack]] — the CRACK MoE on Dankserver
|
||
- [[frank-dragoon]] — the endpoint the Dankclaw ROCmFP4 MTP era became (an Eras row there)
|
||
- [[Quantization by Hardware Architecture]]
|
||
|
||
## Citations
|
||
- SGLang paper (arXiv:2312.07104) — https://arxiv.org/abs/2312.07104 · NeurIPS 2024
|
||
- Repo — https://github.com/sgl-project/sglang · docs — https://docs.sglang.io/
|
||
- AMD / ROCm — https://docs.sglang.io/docs/hardware-platforms/amd_gpu ·
|
||
Structured Outputs — https://docs.sglang.io/docs/advanced_features/structured_outputs
|
||
- lmsysorg/sglang ROCm images — https://hub.docker.com/r/lmsysorg/sglang
|
||
- SGLang vs llama.cpp — https://markaicode.com/vs/sglang-vs-llamacpp/
|