serverAppConfigs/serverAppConfigs-docs/LLM/Radiance Serve Options.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Radiance Serve Options — every flag and the law behind it	Every radiance serve flag grouped by task, with its default and the law that decides when to move it.	llm radiance reference serving	2026-10-30

Radiance serve options

radiance --help prints this table. Every numeric option takes a whole number (not 12k, not auto, unless the option names those); a malformed value exits 2. Upstream main at 2026-10-30; the fleet is pinned older — see the pin note on Radiance Engine.

The one thing to read before any table: the engine derives the VRAM split; no flag sets either pool. The startup receipt prints every term before anything is allocated.

Model, plugins, container

flag	default	meaning
--model PATH	required	a .rad container, a checkpoint served as it stands, or a Hugging Face repo id (owner/name) fetched before startup
--download-dir DIR	$XDG_CACHE_HOME/radiance/hf, else ~/.cache/radiance/hf	where a repo id is fetched
--radiance-home DIRS	compiled prefix	plugin homes, :-separated, first wins
--kernels LIST	every library found	kernel libraries, ,/: separated, first wins; a library not named is not loaded
--served-model-name NAME	container's name	the id /v1/models lists and responses carry; a request may name any model
--override-chat-template PATH	container's	a Jinja template served instead; an unreadable or empty file stops startup
--generation-config PATH	none	sampler defaults from a generation_config.json (else <model>.generation.json, then the container's generation.* metadata)
--control-vector off|on|disabled	disabled	load the GGUF control vector found for the model and start requests off/on; disabled declares no projection and refuses a request that asks
--api-key KEY	none	require Authorization: Bearer KEY or X-Api-Key: KEY, except /health, /v1/health, /ping, the dashboard
--mm-max-patches N	auto (16384 with a vision tower)	patches one encoder pass carries: the largest image, 16×16 px a patch (14×14 on GLM-5.3). 0 serves text only and keeps the tower off the card

Batch shape, context, scheduler

flag	default	meaning
--max-num-batched-tokens N	2048	the largest token count one step may carry; on MoE the knob that sets prefill chunk size, and the activation arena's size
--max-num-seqs N	256	the most sequences one step may carry; the cache's demand, never its funding
--max-model-len N	0 = model's training context	context bound per request. Longer is a 400, never a truncation. Reserves nothing per request; it is the worst case the pool is sized against
--busy-prefill-chunk auto|off|N	auto	a prefill chunk's share of a step that has other work: auto leaves decode rows and waiting requests room, off takes the whole step, N caps it
--prefill-interleave N	0 (off)	decode-only steps after each chunk taken beside decoders, at most 64; buys decoder cadence with the prompt's TTFT
--prefill-span N	1	checkpoint intervals a prefill alone in its step may carry, from its second interval; needs --max-num-batched-tokens of N intervals. Every other step is planned at the shared budget; checkpoints inside a span are not written
--preempt-wait-ms N	5000	how long a preempted prefill waits for the whole-recompute gate before reclaiming blocks and preempting lower-priority work; 0 falls back next step
--rope-scaling SPEC	the checkpoint's	yarn:FACTOR[:ORIG] writes static YaRN over the container's own rotary keys with no reconvert; none takes a checkpoint's YaRN off
--max-queued-requests N	8 × --max-num-seqs	admitted, unfinished requests; past it the answer is 429
--retry-after S	1	the Retry-After a 429 carries

Memory and placement

flag	default	meaning
--placement MODE	auto	all_vram, layer_offload, expert_tiered, auto
--gpu-headroom-mib N	96	VRAM left unclaimed on every card; too little makes the first launch of a never-run kernel fail inside HIP
--host-pool-mib N	0	pinned host tier for weights, per rank; also the KV lever under expert tiering (below)
--host-headroom-mib N	4096	host RAM left free when startup pins memory; a snapshot store's floor takes the shortfall back only down to 1 GiB
--kv-cache-dtype T	bf16	bf16, fp8, and on Qwen3.8-27B/-Flash-Next fp8_v4 (V int4) or k6_v4 (K int6 + V int4). A real quantisation; GLM-5.3 is bf16-only
--state-cache-dtype T	f32	f32 or f16: the recurrent delta-net state cache; f16 halves state pool, checkpoints and traffic and is lossy — qwen35 plugins only
--attn-prefill-fp8 MODE	off	off, qk, pv, both: prefill attention products in E4M3 over an fp8/fp8_v4/k6_v4 cache; lossy, Qwen3.8-27B only where kernels exist
--kv-resident N	0 (off)	stream the sparse attention group's KV from pinned host RAM, keeping an N-token ring on the card. Raised to 32768 and to one decode step's selection; ignored with a warning where unsupported
--embedding-placement ram|vram	ram	token embedding pinned in host memory and read in place, or on the card. A tied head or a host that cannot pin keeps it on the card; an explicit ram refuses instead
--vision-tower-placement vram|ram	vram	encoder towers on the card, or pinned in host memory and read once per image, giving their VRAM to the KV cache
--ngram-placement auto|vram|ram|disk	auto	where an n-gram embedding table lives: all of it on every card, a pinned host copy the card reads, or rows from the container through a row cache. auto takes the first that fits. ram=DIR makes the pinned copy a tmpfs file shared by two engines
--weights-disk-tier	off	read weights that fit neither VRAM nor the host pool from the container with O_DIRECT; every step is paced by the drive
--row-prefetch on|off	on	ask for a host gather's pages before its kernel; output-neutral
--mapped-rewarm on|off	on	at startup, re-ask the page cache for a mapped table's pages an earlier run left behind
--deterministic	off	pin placement to the loader's plan so the resident set cannot move under the heat engine; it does not make output bit-identical across batch positions

KV cache, host tier, checkpoints

flag	default	meaning
--prefix-cache-mib N	0 (off)	copy finished conversations' KV to host memory so a return restores instead of re-prefilling; LRU when full; off while a KV group streams
--prefix-cache-ttl S	3600	seconds an unused cached conversation is kept; 0 keeps it until memory is wanted
--no-prefix-cache	cache on	disable prefix caching
--checkpoint-interval N	2048	linear-attention state checkpoint interval, in tokens; also clamps a hybrid model's prefill chunk
--checkpoint-slots N	auto	snapshots held in total, in pinned host memory; auto = per-policy need for each sequence within an eighth of host RAM; 0 keeps none
--checkpoint-policy NAME	geometric-backoff	also tip-only, keep-all; retention only — writes happen at every interval unless a span skips them
--prefix-resume exact|nearest	exact	exact resumes only where a cold prefill starts a chunk, so a hit answers bit-identically; nearest takes the deepest snapshot, up to a chunk less re-prefilled

Speculation

flag	default	meaning
--num-speculative-tokens N	auto (the drafter's own point; 0 with no head)	speculative window; 0 disables. The ceiling under adaptive
--spec-ngram K	0 (off)	prompt lookup alone, up to K drafts a step from the sequence's own n-grams; leaves a container's draft head out; sets the window to K
--spec-ngram-min N / --spec-ngram-max N	2 / 4 (max 15)	pattern lengths prompt lookup tries, longest first
--draft-window fixed|adaptive	fixed	adaptive follows each sequence's acceptance and the measured step cost
--draft-short-from N	auto	a block drafter that declares a shorter block runs it from N decoding sequences; 0 never; auto = never for a chain, from 2 for a tree
--draft-coupling off|on	off	sample each proposal from the drafter's own shortlist and couple the target's draw to it; same output law, different tokens for a fixed seed, pays at high temperature
--draft-model PATH	—	not implemented at serve: merge the drafter at convert time (rad-convert --draft-model)

Tensor parallelism and collectives

flag	default	meaning
--tp N	1	tensor-parallel ranks; declare divides dimensions by world size and inserts no collective
--tp-wire exact|wht6	exact	wht6 is lossy: rotated 6-bit at 2 ranks, tiered int8 over a two-shot at 3–4, unavailable at 8
--tp-wire-min-kb N	128	smallest all-reduce (KiB) the lossy wire compresses; keeps single-sequence decode exact
--tp-attn-split heads|rows	heads	a prefill step's attention split: rows gives each rank all heads of its rows plus an all-gather; faster where attention dominates, not bit-identical, plugin-gated
--weight-prefetch on|off	on	read a GEMM's weight beside the collective before it; off when another server shares the cards
--p2p auto|on|off	auto	auto asks the driver; on refuses to start without peer access; off exchanges through pinned host memory

LoRA adapters

flag	default	meaning
--lora NAME=PATH	none	serve an adapter: a PEFT directory as model NAME (text), or a .safetensors a request names in loras (image/video). Repeatable
--lora-dir DIR	none	generators: any NAME.safetensors under DIR may be requested
--max-loras N	count of --lora, 2 with --lora-dir alone	adapter slots resident at once
--max-lora-rank R	16 text / 64 generator	slot rank; a larger text adapter is refused, a larger generator combination truncated
--lora-target-modules LIST	the adapters' own	HF module leaf names adapters may target; needed with --max-loras and no --lora

Request defaults and bounds

A request that names a field wins. These set defaults for requests that do not.

flag	default	meaning
--temp F, --top-k N, --top-p F, --min-p F	from the model	sampler defaults; top_k 0 is the 1024 most likely candidates
--typical-p F	1	typical sampling
--presence-penalty F, --frequency-penalty F	0	range [-2, 2]
--repetition-penalty F, --repeat-last-n N	1, 64	-1 looks back over the whole context
--dry-multiplier F, --dry-base F, --dry-allowed-length N, --dry-penalty-last-n N	0, 1.75, 2, -1	DRY; multiplier 0 is off
--dry-sequence-breakers JSON	["\n", ":", "\"", "*"]	at most 16 strings
--xtc-probability F, --xtc-threshold F	0, 0.1	range [0, 1]
--default-max-tokens N	auto	max_tokens for a request that sends none; auto is what the context leaves
--max-tokens-cap N	0 (none)	a larger request is cut to it, not refused; reply ends finish_reason: "length"
--max-n N	max(8, --max-num-seqs)	the largest fan-out n
--max-stop-strings N, --max-stop-bytes N	64, 4096	stop-string limits
--chat-template-kwargs JSON	none	variables for the template on every chat request; a JSON object or @FILE; repeatable
--reasoning-effort S	template's own	default reasoning_effort; passed to the template verbatim, so the legal set is the template's
--reasoning-effort-map MAP	none	FROM=TO,..., checked against the template at startup; none/minimal turn thinking off
--thinking-token-budget N	none	default budget: at N reasoning tokens the block is closed and the answer follows
--reasoning-format auto|none	auto	auto returns reasoning as reasoning_content; none leaves it inline

Server and HTTP

flag	default	meaning
--host ADDR	0.0.0.0	listen address
--port N	8000	listen port
--http-threads N	one per hardware thread, at least 4	workers kept while idle
--read-timeout S	30	a client may go silent this long while sending
--write-timeout S	600	one write to a client may block this long
--keep-alive-timeout S	5	idle connection hold
--max-body-mib N	512	largest body; images and video ride base64 inside it
--no-cors	CORS on	send no CORS headers

Media preprocessing

Each replaces what the container's preprocessor config states, and does nothing for a model that takes no media.

flag	default	meaning
--image-min-pixels N, --image-max-pixels N	container's	the band an image resizes into; one encoder pass still bounds the top
--video-min-pixels N, --video-max-pixels N	container's	same, for a video's sampled frames together
--video-fps F	container's	frames sampled per second
--video-min-frames N, --video-max-frames N	container's	frame-count bounds
--video-max-frame-tokens N	container's	the most prompt tokens one frame becomes; 0 no cap; a token-budget processor takes 70/140/280/560/1120
--image-tokens N	container's	a token-budget processor's image budget (Gemma 4's fixed set; GLM-5.3's ceiling ≤ --mm-max-patches/4)
--max-source-pixels N	67108864	largest input the decoder accepts before resize

Quality measurement (instead of serving)

flag	meaning
--kld-record DIR	run --kld-corpus through prefill and write this model's per-position log-probabilities to DIR: the reference
--kld-ref DIR	score this model against the reference: mean/median/percentile KL and top-1 agreement
--kld-corpus FILE	JSONL, one {"prompt", "score_from"?, "source"?} a line; --kld-record only
--kld-out FILE	the report as JSON; FILE.rows holds every position's numbers
--kld-lora NAME, --kld-lora-sources PREFIX	score through a served adapter, all documents or a source prefix

The mode turns speculation and the prefix cache off and refuses --num-speculative-tokens > 0. See Radiance Conversion for the corpus and the scoring discipline.

Diagnostics

flag	meaning
--debug-graph	print the declared graph, every op's resolved kernel and why, then exit
--debug-selection	the selection table only
--debug-placement	the placement plan only
--debug-tape-dump	the first recorded pass of each kind (TAPEDUMP on stderr, every rank)
--debug-draft-oracle FILE	verify-path test: prompt lookup proposes from FILE with 30 % corrupted drafts; the text must not move
--debug-accept-reference-kernels	serve even with libref rows — not a production configuration
--profile-ops	per-op device timing
--expert-stats FILE	routing counts per layer and step kind, JSON every 30 s and at shutdown
--expert-profile FILE	rank the expert plane by a stats file's decode counts instead of the container profile
--expert-pin-top F	pin the hottest fraction F of routed experts so heat never demotes them
--draft-expert-placement host|auto	pin a draft head's experts in the host pool before any strategy runs
--live	the terminal live view; redirected, one key=value line
-v, -vv	verbose, very verbose

The laws behind the knobs

The split is derived, and the receipt is the referee. Non-expert weights come off the top of the card; the remainder (elastic) is shared by the expert slab and the KV cache. --max-model-len × --max-num-seqs is the cache's demand; the funding is elastic − expert_floor. Under a lending cache (auto/expert_tiered, no --deterministic, VMM present) the cache takes what it can address and lends unused blocks back to experts. Every MiB added to --host-pool-mib lowers the expert floor one-for-one and the cache takes it — until the cache already holds its whole ceiling, where the extra host room shows up as unclaimed. Read the boot line kv: the pool backs N % of the worst case …, never arithmetic.

The activation arena comes off the budget before the split, so both sides pay for --max-num-batched-tokens. While steps are small most of it is lent back to the expert slab; a larger step reclaims it synchronously. That is why a bigger chunk can cost decode residency. On a hybrid model the chunk is clamped to --checkpoint-interval before the token budget, and startup prints which bound applies.

The KV dtype law. bf16 and fp8 are engine-wide; fp8_v4/k6_v4 are narrower layouts whose kernels exist only for particular attention geometries, and the server refuses them elsewhere by plugin name. A drafter's cache stays fp8. Recurrent state and the QSA indexer keep their own widths.

The slot law. A slot costs recurrent state, block tables at full context, snapshot-store RAM and all-reduce rows — never speed by itself. Raising --max-num-seqs raises demand; whether it is funded is the resolver's answer.

The spec law. Measure speculation by step time, not tok/s. Prompt lookup pays where the reply copies its context and costs a verify row elsewhere. adaptive moves the depth only after eight steps and a 2 % gain, because a new window is a new pass shape.

Batch shape is part of the configuration. A different chunk, span or spec depth changes the summation order, so greedy text can differ at a tie. Reproducibility holds per configuration, never across one.

Unknown request fields are ignored; known-but-unimplemented fields are guarded (accepted at their off value, refused otherwise) — the list is on Radiance HTTP API.

Related

• Radiance HTTP API — what the flags expose on the wire.
• Radiance Operations — the container contract and readiness.
• Radiance Plugins — how a flag reaches a plugin (RadBuildCtx).
• frank-archon — the measured deployment of this resolver on two R9700s.