dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(radiance): reference the engine's options, plugins and conversion

Browse Source 
Nine pages under LLM/ document the upstream radiance engine at 8da043c:
- engine shape, plugin model, RAD2 container, ABI and environment
- every serve flag grouped with defaults and the law behind each knob
- the HTTP surface, request fields and refusal classes
- rad-convert, the recipe language, quantisers, imports, reuse, rad-export, KL scoring
- the (arch id, quant) family table and the recipe of record per model
- kernel/quantiser plugin authoring, encodings, build and checks
- architecture plugin authoring: declare, step, probe, lane and media rules
- rad-info/radbench/radtune and the shell harnesses
- the container contract, host prerequisites, readiness and restart triage

Also completes the dangling Related line on the frank-archon endpoint page.
This commit is contained in:
Frank committed 2026-10-30 15:01:37 -07:00
1 parent da5eb968f9
commit 3a04e34414
12 files changed
+2319 -2
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/LLM/Radiance Serve Options.md
+240
View File
@@ -0,0 +1,240 @@
---
type: "reference"
title: "Radiance Serve Options — every flag and the law behind it"
description: "Every radiance serve flag grouped by task, with its default and the law that decides when to move it."
tags:
- llm
- radiance
- reference
- serving
timestamp: 2026-10-30
---
# Radiance serve options
`radiance --help` prints this table. Every numeric option takes a whole number (not `12k`, not
`auto`, unless the option names those); a malformed value exits 2. Upstream `main` at 2026-10-30;
the fleet is pinned older — see the pin note on [[Radiance Engine]].
The one thing to read before any table: **the engine derives the VRAM split; no flag sets either
pool.** The startup receipt prints every term before anything is allocated.
## Model, plugins, container
| flag | default | meaning |
|---|---|---|
| `--model PATH` | required | a `.rad` container, a checkpoint served as it stands, or a Hugging Face repo id (`owner/name`) fetched before startup |
| `--download-dir DIR` | `$XDG_CACHE_HOME/radiance/hf`, else `~/.cache/radiance/hf` | where a repo id is fetched |
| `--radiance-home DIRS` | compiled prefix | plugin homes, `:`-separated, first wins |
| `--kernels LIST` | every library found | kernel libraries, `,`/`:` separated, first wins; a library not named is not loaded |
| `--served-model-name NAME` | container's name | the id `/v1/models` lists and responses carry; a request may name any model |
| `--override-chat-template PATH` | container's | a Jinja template served instead; an unreadable or empty file stops startup |
| `--generation-config PATH` | none | sampler defaults from a `generation_config.json` (else `<model>.generation.json`, then the container's `generation.*` metadata) |
| `--control-vector off\|on\|disabled` | `disabled` | load the GGUF control vector found for the model and start requests off/on; `disabled` declares no projection and refuses a request that asks |
| `--api-key KEY` | none | require `Authorization: Bearer KEY` or `X-Api-Key: KEY`, except `/health`, `/v1/health`, `/ping`, the dashboard |
| `--mm-max-patches N` | `auto` (16384 with a vision tower) | patches one encoder pass carries: the largest image, 16×16 px a patch (14×14 on GLM-5.3). `0` serves text only and keeps the tower off the card |
## Batch shape, context, scheduler
| flag | default | meaning |
|---|---|---|
| `--max-num-batched-tokens N` | 2048 | the largest token count one step may carry; on MoE the knob that sets prefill chunk size, and the activation arena's size |
| `--max-num-seqs N` | 256 | the most sequences one step may carry; the cache's **demand**, never its funding |
| `--max-model-len N` | 0 = model's training context | context bound per request. Longer is a 400, never a truncation. Reserves nothing per request; it is the worst case the pool is sized against |
| `--busy-prefill-chunk auto\|off\|N` | `auto` | a prefill chunk's share of a step that has other work: `auto` leaves decode rows and waiting requests room, `off` takes the whole step, `N` caps it |
| `--prefill-interleave N` | 0 (off) | decode-only steps after each chunk taken beside decoders, at most 64; buys decoder cadence with the prompt's TTFT |
| `--prefill-span N` | 1 | checkpoint intervals a prefill **alone in its step** may carry, from its second interval; needs `--max-num-batched-tokens` of N intervals. Every other step is planned at the shared budget; checkpoints inside a span are not written |
| `--preempt-wait-ms N` | 5000 | how long a preempted prefill waits for the whole-recompute gate before reclaiming blocks and preempting lower-priority work; `0` falls back next step |
| `--rope-scaling SPEC` | the checkpoint's | `yarn:FACTOR[:ORIG]` writes static YaRN over the container's own rotary keys with no reconvert; `none` takes a checkpoint's YaRN off |
| `--max-queued-requests N` | 8 × `--max-num-seqs` | admitted, unfinished requests; past it the answer is 429 |
| `--retry-after S` | 1 | the `Retry-After` a 429 carries |
## Memory and placement
| flag | default | meaning |
|---|---|---|
| `--placement MODE` | `auto` | `all_vram`, `layer_offload`, `expert_tiered`, `auto` |
| `--gpu-headroom-mib N` | 96 | VRAM left unclaimed on every card; too little makes the first launch of a never-run kernel fail inside HIP |
| `--host-pool-mib N` | 0 | pinned host tier for weights, **per rank**; also the KV lever under expert tiering (below) |
| `--host-headroom-mib N` | 4096 | host RAM left free when startup pins memory; a snapshot store's floor takes the shortfall back only down to 1 GiB |
| `--kv-cache-dtype T` | `bf16` | `bf16`, `fp8`, and on Qwen3.8-27B/-Flash-Next `fp8_v4` (V int4) or `k6_v4` (K int6 + V int4). A real quantisation; GLM-5.3 is bf16-only |
| `--state-cache-dtype T` | `f32` | `f32` or `f16`: the recurrent delta-net state cache; f16 halves state pool, checkpoints and traffic and is lossy — `qwen35` plugins only |
| `--attn-prefill-fp8 MODE` | `off` | `off`, `qk`, `pv`, `both`: prefill attention products in E4M3 over an `fp8`/`fp8_v4`/`k6_v4` cache; lossy, Qwen3.8-27B only where kernels exist |
| `--kv-resident N` | 0 (off) | stream the sparse attention group's KV from pinned host RAM, keeping an N-token ring on the card. Raised to 32768 and to one decode step's selection; ignored with a warning where unsupported |
| `--embedding-placement ram\|vram` | `ram` | token embedding pinned in host memory and read in place, or on the card. A tied head or a host that cannot pin keeps it on the card; an explicit `ram` refuses instead |
| `--vision-tower-placement vram\|ram` | `vram` | encoder towers on the card, or pinned in host memory and read once per image, giving their VRAM to the KV cache |
| `--ngram-placement auto\|vram\|ram\|disk` | `auto` | where an n-gram embedding table lives: all of it on every card, a pinned host copy the card reads, or rows from the container through a row cache. `auto` takes the first that fits. `ram=DIR` makes the pinned copy a tmpfs file shared by two engines |
| `--weights-disk-tier` | off | read weights that fit neither VRAM nor the host pool from the container with O_DIRECT; every step is paced by the drive |
| `--row-prefetch on\|off` | on | ask for a host gather's pages before its kernel; output-neutral |
| `--mapped-rewarm on\|off` | on | at startup, re-ask the page cache for a mapped table's pages an earlier run left behind |
| `--deterministic` | off | pin placement to the loader's plan so the resident set cannot move under the heat engine; it does **not** make output bit-identical across batch positions |
## KV cache, host tier, checkpoints
| flag | default | meaning |
|---|---|---|
| `--prefix-cache-mib N` | 0 (off) | copy finished conversations' KV to host memory so a return restores instead of re-prefilling; LRU when full; off while a KV group streams |
| `--prefix-cache-ttl S` | 3600 | seconds an unused cached conversation is kept; `0` keeps it until memory is wanted |
| `--no-prefix-cache` | cache on | disable prefix caching |
| `--checkpoint-interval N` | 2048 | linear-attention state checkpoint interval, in tokens; also clamps a hybrid model's prefill chunk |
| `--checkpoint-slots N` | `auto` | snapshots held **in total**, in pinned host memory; `auto` = per-policy need for each sequence within an eighth of host RAM; `0` keeps none |
| `--checkpoint-policy NAME` | `geometric-backoff` | also `tip-only`, `keep-all`; retention only — writes happen at every interval unless a span skips them |
| `--prefix-resume exact\|nearest` | `exact` | `exact` resumes only where a cold prefill starts a chunk, so a hit answers bit-identically; `nearest` takes the deepest snapshot, up to a chunk less re-prefilled |
## Speculation
| flag | default | meaning |
|---|---|---|
| `--num-speculative-tokens N` | `auto` (the drafter's own point; 0 with no head) | speculative window; 0 disables. The ceiling under `adaptive` |
| `--spec-ngram K` | 0 (off) | prompt lookup alone, up to K drafts a step from the sequence's own n-grams; leaves a container's draft head out; sets the window to K |
| `--spec-ngram-min N` / `--spec-ngram-max N` | 2 / 4 (max 15) | pattern lengths prompt lookup tries, longest first |
| `--draft-window fixed\|adaptive` | `fixed` | `adaptive` follows each sequence's acceptance and the measured step cost |
| `--draft-short-from N` | `auto` | a block drafter that declares a shorter block runs it from N decoding sequences; `0` never; `auto` = never for a chain, from 2 for a tree |
| `--draft-coupling off\|on` | off | sample each proposal from the drafter's own shortlist and couple the target's draw to it; same output law, different tokens for a fixed seed, pays at high temperature |
| `--draft-model PATH` | — | **not implemented** at serve: merge the drafter at convert time (`rad-convert --draft-model`) |
## Tensor parallelism and collectives
| flag | default | meaning |
|---|---|---|
| `--tp N` | 1 | tensor-parallel ranks; declare divides dimensions by world size and inserts no collective |
| `--tp-wire exact\|wht6` | `exact` | `wht6` is lossy: rotated 6-bit at 2 ranks, tiered int8 over a two-shot at 3–4, unavailable at 8 |
| `--tp-wire-min-kb N` | 128 | smallest all-reduce (KiB) the lossy wire compresses; keeps single-sequence decode exact |
| `--tp-attn-split heads\|rows` | `heads` | a prefill step's attention split: `rows` gives each rank all heads of its rows plus an all-gather; faster where attention dominates, not bit-identical, plugin-gated |
| `--weight-prefetch on\|off` | on | read a GEMM's weight beside the collective before it; `off` when another server shares the cards |
| `--p2p auto\|on\|off` | `auto` | `auto` asks the driver; `on` refuses to start without peer access; `off` exchanges through pinned host memory |
## LoRA adapters
| flag | default | meaning |
|---|---|---|
| `--lora NAME=PATH` | none | serve an adapter: a PEFT directory as model NAME (text), or a `.safetensors` a request names in `loras` (image/video). Repeatable |
| `--lora-dir DIR` | none | generators: any `NAME.safetensors` under DIR may be requested |
| `--max-loras N` | count of `--lora`, 2 with `--lora-dir` alone | adapter slots resident at once |
| `--max-lora-rank R` | 16 text / 64 generator | slot rank; a larger text adapter is refused, a larger generator combination truncated |
| `--lora-target-modules LIST` | the adapters' own | HF module leaf names adapters may target; needed with `--max-loras` and no `--lora` |
## Request defaults and bounds
A request that names a field wins. These set defaults for requests that do not.
| flag | default | meaning |
|---|---|---|
| `--temp F`, `--top-k N`, `--top-p F`, `--min-p F` | from the model | sampler defaults; `top_k` 0 is the 1024 most likely candidates |
| `--typical-p F` | 1 | typical sampling |
| `--presence-penalty F`, `--frequency-penalty F` | 0 | range [-2, 2] |
| `--repetition-penalty F`, `--repeat-last-n N` | 1, 64 | `-1` looks back over the whole context |
| `--dry-multiplier F`, `--dry-base F`, `--dry-allowed-length N`, `--dry-penalty-last-n N` | 0, 1.75, 2, -1 | DRY; multiplier 0 is off |
| `--dry-sequence-breakers JSON` | `["\n", ":", "\"", "*"]` | at most 16 strings |
| `--xtc-probability F`, `--xtc-threshold F` | 0, 0.1 | range [0, 1] |
| `--default-max-tokens N` | `auto` | `max_tokens` for a request that sends none; `auto` is what the context leaves |
| `--max-tokens-cap N` | 0 (none) | a larger request is **cut to it**, not refused; reply ends `finish_reason: "length"` |
| `--max-n N` | max(8, `--max-num-seqs`) | the largest fan-out `n` |
| `--max-stop-strings N`, `--max-stop-bytes N` | 64, 4096 | stop-string limits |
| `--chat-template-kwargs JSON` | none | variables for the template on every chat request; a JSON object or `@FILE`; repeatable |
| `--reasoning-effort S` | template's own | default `reasoning_effort`; passed to the template verbatim, so the legal set is the template's |
| `--reasoning-effort-map MAP` | none | `FROM=TO,...`, checked against the template at startup; `none`/`minimal` turn thinking off |
| `--thinking-token-budget N` | none | default budget: at N reasoning tokens the block is closed and the answer follows |
| `--reasoning-format auto\|none` | `auto` | `auto` returns reasoning as `reasoning_content`; `none` leaves it inline |
## Server and HTTP
| flag | default | meaning |
|---|---|---|
| `--host ADDR` | 0.0.0.0 | listen address |
| `--port N` | 8000 | listen port |
| `--http-threads N` | one per hardware thread, at least 4 | workers kept while idle |
| `--read-timeout S` | 30 | a client may go silent this long while sending |
| `--write-timeout S` | 600 | one write to a client may block this long |
| `--keep-alive-timeout S` | 5 | idle connection hold |
| `--max-body-mib N` | 512 | largest body; images and video ride base64 inside it |
| `--no-cors` | CORS on | send no CORS headers |
## Media preprocessing
Each replaces what the container's preprocessor config states, and does nothing for a model that
takes no media.
| flag | default | meaning |
|---|---|---|
| `--image-min-pixels N`, `--image-max-pixels N` | container's | the band an image resizes into; one encoder pass still bounds the top |
| `--video-min-pixels N`, `--video-max-pixels N` | container's | same, for a video's sampled frames together |
| `--video-fps F` | container's | frames sampled per second |
| `--video-min-frames N`, `--video-max-frames N` | container's | frame-count bounds |
| `--video-max-frame-tokens N` | container's | the most prompt tokens one frame becomes; `0` no cap; a token-budget processor takes 70/140/280/560/1120 |
| `--image-tokens N` | container's | a token-budget processor's image budget (Gemma 4's fixed set; GLM-5.3's ceiling ≤ `--mm-max-patches`/4) |
| `--max-source-pixels N` | 67108864 | largest input the decoder accepts before resize |
## Quality measurement (instead of serving)
| flag | meaning |
|---|---|
| `--kld-record DIR` | run `--kld-corpus` through prefill and write this model's per-position log-probabilities to DIR: the reference |
| `--kld-ref DIR` | score this model against the reference: mean/median/percentile KL and top-1 agreement |
| `--kld-corpus FILE` | JSONL, one `{"prompt", "score_from"?, "source"?}` a line; `--kld-record` only |
| `--kld-out FILE` | the report as JSON; `FILE.rows` holds every position's numbers |
| `--kld-lora NAME`, `--kld-lora-sources PREFIX` | score through a served adapter, all documents or a `source` prefix |
The mode turns speculation and the prefix cache off and refuses `--num-speculative-tokens > 0`.
See [[Radiance Conversion]] for the corpus and the scoring discipline.
## Diagnostics
| flag | meaning |
|---|---|
| `--debug-graph` | print the declared graph, every op's resolved kernel and why, then exit |
| `--debug-selection` | the selection table only |
| `--debug-placement` | the placement plan only |
| `--debug-tape-dump` | the first recorded pass of each kind (`TAPEDUMP` on stderr, every rank) |
| `--debug-draft-oracle FILE` | verify-path test: prompt lookup proposes from FILE with 30 % corrupted drafts; the text must not move |
| `--debug-accept-reference-kernels` | serve even with `libref` rows — not a production configuration |
| `--profile-ops` | per-op device timing |
| `--expert-stats FILE` | routing counts per layer and step kind, JSON every 30 s and at shutdown |
| `--expert-profile FILE` | rank the expert plane by a stats file's `decode` counts instead of the container profile |
| `--expert-pin-top F` | pin the hottest fraction F of routed experts so heat never demotes them |
| `--draft-expert-placement host\|auto` | pin a draft head's experts in the host pool before any strategy runs |
| `--live` | the terminal live view; redirected, one `key=value` line |
| `-v`, `-vv` | verbose, very verbose |
## The laws behind the knobs
**The split is derived, and the receipt is the referee.** Non-expert weights come off the top of
the card; the remainder (`elastic`) is shared by the expert slab and the KV cache. `--max-model-len`
× `--max-num-seqs` is the cache's **demand**; the funding is `elastic − expert_floor`. Under a
lending cache (`auto`/`expert_tiered`, no `--deterministic`, VMM present) the cache takes what it
can address and lends unused blocks back to experts. **Every MiB added to `--host-pool-mib` lowers
the expert floor one-for-one and the cache takes it** — until the cache already holds its whole
ceiling, where the extra host room shows up as unclaimed. Read the boot line
`kv: the pool backs N % of the worst case …`, never arithmetic.
**The activation arena comes off the budget before the split**, so both sides pay for
`--max-num-batched-tokens`. While steps are small most of it is lent back to the expert slab; a
larger step reclaims it synchronously. That is why a bigger chunk can cost decode residency. On a
hybrid model the chunk is clamped to `--checkpoint-interval` before the token budget, and startup
prints which bound applies.
**The KV dtype law.** `bf16` and `fp8` are engine-wide; `fp8_v4`/`k6_v4` are narrower layouts whose
kernels exist only for particular attention geometries, and the server refuses them elsewhere by
plugin name. A drafter's cache stays fp8. Recurrent state and the QSA indexer keep their own widths.
**The slot law.** A slot costs recurrent state, block tables at full context, snapshot-store RAM
and all-reduce rows — never speed by itself. Raising `--max-num-seqs` raises demand; whether it is
funded is the resolver's answer.
**The spec law.** Measure speculation by step time, not tok/s. Prompt lookup pays where the reply
copies its context and costs a verify row elsewhere. `adaptive` moves the depth only after eight
steps and a 2 % gain, because a new window is a new pass shape.
**Batch shape is part of the configuration.** A different chunk, span or spec depth changes the
summation order, so greedy text can differ at a tie. Reproducibility holds per configuration, never
across one.
**Unknown request fields are ignored; known-but-unimplemented fields are guarded** (accepted at
their off value, refused otherwise) — the list is on [[Radiance HTTP API]].
## Related
- [[Radiance HTTP API]] — what the flags expose on the wire.
- [[Radiance Operations]] — the container contract and readiness.
- [[Radiance Plugins]] — how a flag reaches a plugin (`RadBuildCtx`).
- [[frank-archon]] — the measured deployment of this resolver on two R9700s.