Frank
a5fe045b58
Structural repair: - 13 pages were the same document twice (each exactly 2x its HEAD size), a second frontmatter block whose closing fence had been glued onto a body line by a frontmatter rewrite that re-read its own output. Un-doubled by cutting at the second frontmatter and keeping the copy whose body is a superset; the orphan census reports 0 lost tokens against HEAD. - log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first — 66 dated headings merged into 34, titles demoted to bold leads, every content line verified present. The file had been descending at the head and ascending at the tail. Ownership moves: - New `LLM/frank-medium.md` is the locator for the retired Dankserver endpoint: era windows quoted from each era page's own banner, no numbers restated, and the two unresolved date conflicts recorded instead of silently resolved. `frank-archon.md` links to it instead of restating the pre-rename window. - The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone; `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor verdicts and cross-split spec decode, and the four links that pointed at the stubs were repointed at it or at the drafter page. - Authentik's three stub pages fold into `Authentik.md`, with the process detail and the outpost reference split out so the parent stays under the length hint. Verified corrections (host census + live probe, 2026-10-19): - The MXFP4 checkpoint two pages called the "on-disk rollback" is gone: /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on the era page that owns that payload; the swap page now links to it. - frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2 arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run census is kept once (155 requests) instead of twice (~120 and 155). - Fleet shapes re-confirmed against the live endpoints before touching any page: 3x200000 fp8 KV, 2x200704, 2x262144. Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
155 lines
8.5 KiB
Markdown
155 lines
8.5 KiB
Markdown
---
|
||
type: reference
|
||
title: SGLang
|
||
description: "What SGLang is, how it schedules and caches, how it compares to the runtimes here, and whether it fits this fleet."
|
||
tags:
|
||
- sglang
|
||
- serving
|
||
- inference
|
||
- structured-output
|
||
- radix-attention
|
||
- llama.cpp
|
||
timestamp: 2026-09-01
|
||
---
|
||
|
||
# SGLang
|
||
|
||
SGLang (for **S**tructured **G**eneration **Lang**uage) is a high-throughput
|
||
serving engine for large (and vision-language) models, from the LMSYS / UC
|
||
Berkeley team, published at NeurIPS 2024. Today it is one of the two dominant
|
||
open-source serving engines, alongside vLLM. People talk about it because it
|
||
makes "call the model, branch, force the output into JSON" programs *fast*.
|
||
|
||
## What it is — in one paragraph
|
||
|
||
You serve an LLM by writing a small program: call the model, then based on its
|
||
output call it again, branch into parallel attempts, and force the answer into a
|
||
JSON schema. SGLang co-designs a Python-embedded language with a runtime that can
|
||
*see* the program's structure, then exploits it: it caches and reuses the model's
|
||
"working memory" (the KV cache) across requests that share a prompt prefix,
|
||
batches new requests into running generations without waiting for a batch to
|
||
drain, and forces tightly-structured output (JSON / regex / tool tags) by masking
|
||
illegal tokens one step at a time — or several at once. Reported payoff:
|
||
**up to 6.4× higher throughput** on structured / agentic / RAG workloads vs vLLM,
|
||
TGI, and similar.
|
||
|
||
## Core architecture
|
||
|
||
Two halves that can also run independently:
|
||
- **Frontend** — a Python DSL (`@sgl.function`, `gen`/`select`/`fork`/`join`/
|
||
`extend`) for multi-call LLM programs with branching, parallel sampling, tool
|
||
calls, and structured output.
|
||
- **Backend runtime** — a high-throughput server (`python -m sglang.launch_server`)
|
||
or an in-process `Engine`. OpenAI-compatible `/v1`, so RadixAttention and
|
||
structured decoding work with a vanilla OpenAI client.
|
||
|
||
Building blocks:
|
||
- **z-normalization** — pre-compile per-request sampling metadata into contiguous
|
||
tensors so the GPU sampler does one bulk copy per batch per step (kills the
|
||
CPU scheduler bottleneck).
|
||
- **PagedTensor** — the KV cache is paged (fixed-size pages) and laid out
|
||
non-contiguously in a global pool; mirrors vLLM's PagedAttention.
|
||
- **RadixAttention** — a radix tree of shared token prefixes; on a new request it
|
||
walks to the longest cached prefix, reuses those KV blocks, and only prefills
|
||
the divergent suffix. Payoff is compute + throughput for long shared prefixes
|
||
(RAG, agents), not raw memory.
|
||
- **Overlap scheduler** — runs the CPU prep for step t+1 *under* the GPU forward
|
||
of step t, so kernels launch back-to-back with no idle bubble (plus CUDA
|
||
graphs + torch.compile).
|
||
- **Structured / output-guided generation** — compiles the constraint (JSON
|
||
schema / regex / EBNF) into a finite-state machine over the vocabulary and masks
|
||
disallowed logits; **jump-forward** decoding emits forced token runs in one step.
|
||
Backends: XGrammar (default), Outlines.
|
||
|
||
## SGLang vs llama.cpp vs vLLM
|
||
|
||
| Dimension | SGLang | llama.cpp |
|
||
|-----------|--------|-----------|
|
||
| Runtime | Python + CUDA/C++ kernels | C/C++, no Python required |
|
||
| GPU requirement | GPU required, **no CPU path** | CPU-only / GPU-only / hybrid offload |
|
||
| Model format | HF safetensors; AWQ/GPTQ/FP8 on the fly; **GGUF via llama.cpp backend** | **GGUF only** (convert first) |
|
||
| KV reuse | RadixAttention (always-on radix tree) | basic prompt caching, no radix sharing |
|
||
| Structured output | first-class (XGrammar, jump-forward) | narrower |
|
||
| Concurrency | continuous batching + overlap scheduler | single-node, lower concurrency |
|
||
| Multi-GPU | native tensor parallelism | layer-split, more manual |
|
||
| Setup | `pip install sglang[all]`, venv, CUDA match | static binary / Docker, minutes |
|
||
| License | Apache 2.0 | MIT |
|
||
|
||
**Pick SGLang** when you need many concurrent users, structured / JSON / tool-call
|
||
output, or repeated long prefixes (RAG, agents) where RadixAttention pays off, and
|
||
you own a GPU + Python/CUDA dependency chain. **Pick llama.cpp** for the fastest
|
||
download-to-endpoint path, CPU/edge, GGUF libraries, and quantization breadth.
|
||
**Pick vLLM** for the broadest model-hardware coverage and mature multi-GPU.
|
||
|
||
The paper's headline 6.4× / 5× numbers are vs **vLLM v0.1** (no prefix caching);
|
||
the gap narrows vs modern vLLM — benchmark on your own traffic.
|
||
|
||
## Deployment model & gotchas
|
||
- **Pluggable backends:** default vLLM engine, SGLang's own `sglang` engine, or a
|
||
recent beta **`llama.cpp` backend** that serves GGUF and reuses `llama-server`.
|
||
- **VRAM headroom runs higher** than llama.cpp (pre-allocated KV pool + radix
|
||
tree). Size with `--mem-fraction-static`, `--context-length`,
|
||
`--chunked-prefill-size`. OOM → lower context / fraction or a smaller quant.
|
||
- **Model-format lock-in:** wants safetensors + AWQ/GPTQ/FP8 (or GGUF via the
|
||
llama.cpp backend). The current CPU-Q6_K_P and ROCmFP4-MTP models are **GGUF**.
|
||
- **AMD / ROCm: yes, officially.** Prebuilt `lmsysorg/sglang` ROCm Docker image or
|
||
`pip`. Quant on AMD: FP8 (Aiter/Triton), AWQ (Triton), MXFP4 (CDNA3/4),
|
||
`petit_nvfp4`. Marlin-based methods are **out** on AMD. Kernel work targets
|
||
**datacenter Instinct (CDNA2/3/4)** — consumer/client parts like **Strix Halo
|
||
(gfx1151)** are supported at the ROCm level but are not the kernel-tuning target.
|
||
- **Prefix diversity kills RadixAttention** (per-user/personalized prompts → tree
|
||
degenerates). Single-replica trees need prefix-aware routing across a fleet
|
||
(`sglang-router --policy cache_aware`). Fast-moving API — pin versions.
|
||
|
||
## Relevance to our servers
|
||
|
||
### Dankserver — NVIDIA RTX 3090 (24 GB) + 62 GB RAM — 35B MoE
|
||
A **genuine fit candidate (with a migration step)**. The 3090 is CUDA-native.
|
||
A 35B MoE (~3B active) is ideal for continuous batching + structured generation,
|
||
and 24 GB can hold a quantized 35B MoE (FP8/AWQ safetensors ~18–20 GB) with room
|
||
for KV. MoE expert parallelism is supported.
|
||
- **What changes:** the current model is a **CPU GGUF (Q6_K_P)**; SGLang wants
|
||
safetensors (FP8/AWQ) for CUDA, OR use SGLang's **`llama.cpp` backend** to serve
|
||
the existing GGUF under SGLang's frontend. Check for an AWQ/FP8 variant of the
|
||
HauhauCS checkpoint on Hugging Face.
|
||
- **Where it wins:** repeated structured/JSON/tool calls or a long shared system
|
||
prompt (RAG) → RadixAttention + jump-forward give big gains vs the current
|
||
sequential CPU pipeline. Simple low-concurrency chat → the operational weight
|
||
may not pay off.
|
||
|
||
### Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified) — Qwen3.8-27B ROCmFP4 MTP
|
||
A **niche / experimental fit, not a drop-in replacement**. 64 GiB is generous and
|
||
SGLang-on-ROCm is supported (Docker/pip), but:
|
||
- Our model is a **custom ROCmFP4/MTP GGUF**. SGLang's primary path wants
|
||
safetensors + an AMD quant (FP8/AWQ); MXFP4 needs CDNA3/4 and Strix Halo (CDNA
|
||
3.5 *client*) is **not** an SGLang/AMD kernel target. Quant availability for this
|
||
exact MTP model is unverified, and we'd lose the bespoke MTP draft-head speedup.
|
||
- **Best-fit path:** SGLang's **`llama.cpp` backend** on top of our existing ROCm
|
||
llama.cpp / GGUF — gain the structured-output DSL, RadixAttention, and scheduler
|
||
without rebuilding the kernel stack. Or test the native `lmsysorg/sglang` ROCm
|
||
image as a proof-of-concept. Treat it as an enhancement/experiment, not a
|
||
production swap, until a quant for the MTP model exists and perf is measured on
|
||
gfx1151.
|
||
|
||
## Tradeoffs at a glance
|
||
- Higher baseline VRAM than llama.cpp (KV pool + radix tree).
|
||
- GPU/accelerator required — no pure-CPU path (llama.cpp is the only option of the
|
||
trio for a CPU-only box).
|
||
- Model-format lock-in (safetensors/AWQ/GPTQ/FP8, or GGUF via llama.cpp backend).
|
||
- Marlin-based quant out on AMD; Strix Halo is not an MXFP4 target.
|
||
- Prefix diversity can nullify RadixAttention; pin versions; watch cache-report.
|
||
|
||
## See Also
|
||
- [[Llama Cpp Runtimes]] / [[Llama Cpp Code Diff]] — the llama.cpp side of the trio
|
||
- [[dealign-ornith-crack]] — the CRACK MoE on Dankserver
|
||
- [[frank-dragoon]] — the endpoint the Dankclaw ROCmFP4 MTP era became (an Eras row there)
|
||
- [[Quantization by Hardware Architecture]]
|
||
|
||
## Citations
|
||
- SGLang paper (arXiv:2312.07104) — https://arxiv.org/abs/2312.07104 · NeurIPS 2024
|
||
- Repo — https://github.com/sgl-project/sglang · docs — https://docs.sglang.io/
|
||
- AMD / ROCm — https://docs.sglang.io/docs/hardware-platforms/amd_gpu ·
|
||
Structured Outputs — https://docs.sglang.io/docs/advanced_features/structured_outputs
|
||
- lmsysorg/sglang ROCm images — https://hub.docker.com/r/lmsysorg/sglang
|
||
- SGLang vs llama.cpp — https://markaicode.com/vs/sglang-vs-llamacpp/
|