Live-verified post-deploy: context_len=262144, max_running_requests=2,
max_total_num_tokens=606208 in the boot log; a 200,000-token prompt
(compaction height is 183,501) prefilled+completed in one request
(97.3 s, ~2,056 tok/s eager); two concurrent 50k requests answered.
Docs-only change.
The KV pool stays pinned at 606,208 tokens (card is full: 531 MiB free at
the old shape; autofit's 687,388 would leave 0.53 GB and OOM on the first
long request). 2 x 262,144 = 524,288 leaves ~82k tokens of pool slack.
Motivation: harness compaction. frank-primary's contextWindow 262,144
compacts at 70% = 183,501 tokens, and the compaction request carries a
32,768-token summary budget -> 216,269 must fit ONE frank-small request.
The 151,552 slot window could not hold it; the native window does.
No spec-decode interaction: checkpoint ships zero mtp.* tensors (config
lies), seam keeps every --speculative-* flag away from the engine.
Replaces llama.cpp Gemma4-12B-QAT (payload + binary retained as rollback).
SGLang in a self-contained python3.12 venv with an extracted CUDA 13.0
nvcc toolchain for the sgl-kernel/flashinfer JIT.
Shape: fp8 KV, pinned 606,208-token pool / 4 x 151,552 slots, bf16 mamba
states pinned at 32 slots (the engine autofills the ssm pool to a fixed
~5.1 GB regardless of dtype), eager prefill (no headroom for graph
capture), no spec decode (the checkpoint ships no mtp.* tensors despite
mtp_num_hidden_layers=1 — NEXTN drafts with an uninitialised 4.24 GiB
ghost at 0% acceptance).
Verified: 83.9 tok/s decode, 170k single prompt, 4 concurrent 54k
requests, vision (in-checkpoint tower), reasoning_content field.
Frank