Merge pull request 'docs(radiance): reference the engine — options, plugins, conversion, tools, operations' (#88) from docs/radiance-engine-reference into main
This commit was merged in pull request #88.
This commit is contained in:
dank committed
commit
b45771e87e
12 files changed
+2320
-2
No files matched your search
@@ -0,0 +1,287 @@
|
||||
---
|
||||
type: "howto"
|
||||
title: "Radiance Architecture Plugins — declaring a model"
|
||||
description: "Writing a radiance architecture plugin: declaring weights, buffers, KV groups and ops, then issuing the step."
|
||||
tags:
|
||||
- llm
|
||||
- radiance
|
||||
- plugins
|
||||
- howto
|
||||
timestamp: 2026-10-30
|
||||
---
|
||||
|
||||
# Radiance architecture plugins
|
||||
|
||||
An architecture plugin declares a model in two phases: `rad_arch_declare` names every weight,
|
||||
buffer, KV group and op **once**, against the model's metadata; `rad_arch_step` issues the ops for
|
||||
one step. The engine picks the plugin by the exact `(rad_arch_id, rad_arch_quant)` pair — no
|
||||
fallback — so one family at two weight formats is two `.so` files, and branching on a quant
|
||||
descriptor inside one plugin is the anti-pattern the split exists to prevent.
|
||||
|
||||
The in-tree plugins are the templates: `arch/llama_fp8/llama_fp8.cpp` (the smallest complete
|
||||
decoder), `arch/gemma3`, `arch/qwen35_bf16`, and the header-only blocks in `arch/common/`
|
||||
(`rad_block_attn*.h`, `rad_block_mlp*.h`, `rad_block_moe_fp8.h`, `rad_block_gdn*.h`,
|
||||
`rad_block_dflash2.h`, `rad_block_mtp*.h`, `rad_block_vit.h`, `rad_block_ple.h`,
|
||||
`rad_block_glm_*.h`, `rad_qsa.h`, …). Include those by `<arch/rad_arch.h>` — they are installed
|
||||
beside the ABI and the in-tree plugins include them by that spelling.
|
||||
|
||||
## Exports
|
||||
|
||||
```c
|
||||
const char* rad_arch_id(void); /* "llama", "qwen35", ... */
|
||||
const char* rad_arch_quant(void); /* "" unquantised, else the descriptor served */
|
||||
int rad_arch_serves_imports(void); /* optional: 0 declines an imported quantisation */
|
||||
int rad_arch_probe(const RadModelMeta* meta, RadArchProbe* out); /* optional */
|
||||
int rad_arch_declare(RadBuilder* b, const RadModelMeta* meta, const RadBuildCtx* ctx);
|
||||
void rad_arch_step(struct RadCtx* c, const struct RadBatch* batch);
|
||||
const RadChatFormat* rad_arch_chat_format(const RadModelMeta* meta); /* optional */
|
||||
const RadAudioFormat* rad_arch_audio_format(const RadModelMeta* meta); /* optional */
|
||||
```
|
||||
|
||||
Everything but declare/step is optional. `RAD_ARCH_PLUGIN(NS, ID, QUANT, VERSION, DESC)` in
|
||||
`arch/common/rad_arch.h` emits the whole export block, including `rad_plugin_info` (kind
|
||||
`RAD_PLUGIN_ARCH`). The plugin's **name must be unique**; the architecture id may repeat.
|
||||
|
||||
`rad_arch_serves_imports` exists because a checkpoint whose quantisation was imported weight by
|
||||
weight (Quark, ParoQuant, GPTQ, NVFP4) names its producing tool, which no plugin claims: the family's
|
||||
one quantised plugin serves it, and two are refused by name. A second quantised plugin that reads
|
||||
only its own containers returns 0.
|
||||
|
||||
## Declare: the metadata
|
||||
|
||||
`RadModelMeta` is what the loader read from the container's header before any plugin ran: `arch_id`,
|
||||
`name`, `quant`, `n_layers`, `n_embd`, `n_head`, `n_head_kv`, **`head_dim`**, `n_ff`, `n_vocab`,
|
||||
`n_ctx_train`, expert counts, `rms_eps`, `rope_theta`, and a free-form `kv_key`/`kv_val` table for
|
||||
everything else (`rad_meta_geti/getf/gets`). Read dimensions from here — one plugin covers every size
|
||||
in a family.
|
||||
|
||||
**`head_dim` is read, never derived.** On the Qwen hybrids it is 256 while `hidden / heads` is 213;
|
||||
a derived value silently breaks projection widths, KV page size and the attention kernel's band.
|
||||
|
||||
`RadBuildCtx` is the deployment the plugin must declare against:
|
||||
|
||||
| field | meaning |
|
||||
|---|---|
|
||||
| `rank`, `world_size` | declare runs once per rank; **dimensions are already divided** — the core inserts no collectives |
|
||||
| `max_tok`, `max_seqs`, `max_ctx`, `max_spec` | the step bounds: buffer sizing, kernel bands, the scheduler's window |
|
||||
| `max_mixed_tok` | the most a multi-sequence step may carry; a wider step is one sequence (prefill span) |
|
||||
| `tp_wire_lossy`, `tp_wire_min_bytes` | invert into `all_reduce`'s `exact`; below the floor a message must go exact |
|
||||
| `kv_dtype`, `kv_quant` | the attention cache width and, under fp8, its narrow layout (`RAD_KVQ_V4`, `RAD_KVQ_K6V4`) |
|
||||
| `attn_prefill_fp8` | `RAD_ATTN_FP8_*` bits to pass as `attn_paged`'s `prefill_fp8` |
|
||||
| `kv_resident_blocks`, `kv_stream_ok` | KV streaming: the ring depth, and permission to declare the streamed-only ops |
|
||||
| `state_dtype` | `RAD_F16` when `--state-cache-dtype f16` is served by this plugin |
|
||||
| `is_draft`, `scope` | a drafter/encoder declare; `scope` prefixes declared names (`draft`, `vision`, `audio`) |
|
||||
| `shape_probe` | a **sizing** declare at a smaller `max_tok`; must not touch state the real declare wrote |
|
||||
| `max_enc_patches`, `max_audio_frames` | the encoder pass sizes |
|
||||
| `max_out_rows` | logits rows the KL mode needs (0 = the plugin's own rule) |
|
||||
| `cvec` | the loaded control vector, or null |
|
||||
| `tp_attn_rows` | row-split prefill attention requested |
|
||||
| `draft_coupling`, `draft_tree` | declare the coupled/tree forms |
|
||||
| `weight_prefetch_off` | declare no weight prefetch beside collectives |
|
||||
|
||||
## Declare: weights
|
||||
|
||||
```c
|
||||
static const RadWeightDecl d = {
|
||||
.dtype = RAD_BF16, .rank = 2, .shape = {n_embd, n_ff},
|
||||
.access = RAD_ACCESS_PER_TOKEN, .shard = RAD_SHARD_ROW,
|
||||
.group = grp_layer(l), .optional = 0,
|
||||
.n_row_parts = 0, .source = nullptr, .planes = nullptr,
|
||||
};
|
||||
rad_weight w = rad_decl_weight(b, "blk.0.mlp.gate.weight", &d);
|
||||
```
|
||||
|
||||
- **`access`** tells the placement planner how the weight is read: `PER_TOKEN` (dense),
|
||||
`PER_REQUEST` (an encoder tower, read once per request), `CONDITIONAL` (routed experts),
|
||||
`RARE`, `VOCAB` (lm_head/embedding: read every token but huge, and one is a gather).
|
||||
- **`shard`**: `NONE` (replicated), `ROW` (split dim 0), `COL` (split dim 1, needs a following
|
||||
all-reduce). The plugin says; the core does not infer. Uneven splits use
|
||||
`rad_weight_shard_span`/`_parts` (a width that does not divide into whole 128-column blocks; the
|
||||
delta net's `[q|k|v|z]` of unequal part widths).
|
||||
- **`group`** is the unit of movement (`layer`, `expert`, `slot`); an expert's gate/up/down share one,
|
||||
because the unit of transfer is an expert, not a tensor.
|
||||
- **`row_parts`** describes what a `SHARD_ROW` weight stacks along dim 0 (`[gate|up]`,
|
||||
`[q|k|v|z]`). **A tensor-parallel rank owns a slice of each stacked projection**, never a
|
||||
contiguous slice of the stack; the parts cannot be inferred from a fused container tensor, and a
|
||||
contiguous slice produces fluent wrong text.
|
||||
- **`source`/`planes`** view an existing stored weight: a block-fp8 linear's `codes,scale` are one
|
||||
logical weight taken by a GEMM as two operands. Null `source` is the weight of its own name; null
|
||||
`planes` takes every plane.
|
||||
- A dtype other than the plane's is legal only as an exact widening (a bf16 norm read as f32).
|
||||
|
||||
The **name map** (`rad_decl_name_map`) is the plugin's map from checkpoint tensor names to declared
|
||||
weight names, and it is the only place convert-time fusions live: `COPY`, `CONCAT`, `WNORM` (a
|
||||
weight-normalised pair folded to the plain weight), `ROWS`, `TRANSPOSE`, `EMA` (an EnCodec codebook),
|
||||
`TAP`, `COLS`. `rad-convert` reads this map from the plugin, so it needs no knowledge of the model.
|
||||
`RAD_MAP_COPY` with several sources means "first present wins" (a tied lm_head); `CONCAT` requires
|
||||
all. Ask for a weight's encoding with `rad_weight_encoding` **after** its map entry when the graph
|
||||
depends on the format (fp8 GEMM vs plain, rotated expert kernels).
|
||||
|
||||
## Declare: buffers and KV groups
|
||||
|
||||
`rad_decl_buffer` names every activation: `RAD_BUF_TRANSIENT` (liveness-analysed, may share bytes),
|
||||
`RAD_BUF_PERSIST` (across steps: a calibration accumulator, a routing histogram), or
|
||||
`RAD_BUF_DERIVED` (computed once per step by the core from the batch — `derive` names the
|
||||
derivation, e.g. `conv_state_index`). Buffers live in one arena sized once, not an allocator per
|
||||
step.
|
||||
|
||||
`rad_decl_kv_group` declares each state kind: `RAD_KV_FULL` (paged per-token, growing),
|
||||
`RAD_KV_WINDOW` (bounded by a window), `RAD_KV_LINEAR` (per-sequence fixed state, GDN),
|
||||
`RAD_KV_CONV` (a rolling window of `width-1+num_spec`), `RAD_KV_RING` (window floor set at runtime).
|
||||
The **block size is not declared** — it comes from the resolved attention kernel's constraints; read
|
||||
it back with `rad_kv_block_size`. `rad_bind_layer_kv` binds each layer so the block manager knows how
|
||||
many state instances a sequence owns; `rad_kv_group_zero` (a rank with no heads),
|
||||
`rad_kv_group_replicated` (every rank holds the same bytes), and `rad_kv_group_dense_read` (read over
|
||||
the whole history every step, so it cannot stream) are the three refinements.
|
||||
|
||||
## Declare: ops
|
||||
|
||||
```c
|
||||
rad_op h = RAD_OP(b, "gemm_w4a8_nt",
|
||||
RAD_PARAMS(RAD_RANGE("M", 1, max_tok), RAD_INT("N", 8192), RAD_INT("K", 5120)),
|
||||
RAD_WEIGHTS(w));
|
||||
rad_op_reads (b, h, RAD_BUFS(b_x), n);
|
||||
rad_op_writes(b, h, RAD_BUFS(b_y), n);
|
||||
```
|
||||
|
||||
- Parameter names and operand order come from `docs/OPS.md`; a new name is allowed only when the
|
||||
vocabulary has none, and the first plugin to declare it fixes its schema.
|
||||
- `RAD_RANGE` on exactly one parameter per op gives the selector its bucket table. The core carries
|
||||
every parameter opaquely except `RAD_PROLE_SEQ_CHUNK` (cut steps on it) and `RAD_PROLE_CAPACITY`.
|
||||
- `rad_op_reads`/`rad_op_writes` are the buffer planner's liveness. An undeclared use is assumed live
|
||||
for the whole program: sound but wasteful.
|
||||
- **Fusion is selection.** Ask for the fused op; if `rad_op_resolved` is false, emit the chain. A
|
||||
fusion's `replaces` hook is a checkable equivalence, not a rewrite.
|
||||
- `rad_op_declared(b, "op")` asks whether any library declares an optional op; use it before
|
||||
emitting a fused form only an optional library defines.
|
||||
- Buffers touched from the **second lane** go through `rad_buf_concurrent`, and an architecture whose
|
||||
lane-1 work lies in one stretch declares `rad_lane1_window_begin/end` around it. The runtime
|
||||
refuses a lane-1 op outside the window; the window lets buffers used only outside it share bytes.
|
||||
|
||||
Step output: `rad_declare_logits` names the buffer the sampler reads (its dim 0 bounds `n_out`; a
|
||||
[n_tok, vocab] logits buffer at a large vocabulary is gigabytes — declare the gather +
|
||||
`logits_gemm` shape instead). An embedding program declares `rad_declare_pooled` instead (one row a
|
||||
sequence, no sampler, no generation). `rad_declare_scores` opts into host-side scored steps.
|
||||
|
||||
## Declare: drafters, encoders, generators
|
||||
|
||||
- `rad_declare_drafter(RadDrafterDecl)` — depth, block, proposal shape. `rad_drafter_shared_kv` (a
|
||||
drafter with no cache of its own), `rad_drafter_tree` (a tree over the proposals),
|
||||
`rad_drafter_short_block` (a second, shorter pass) are refinements. Declare the drafter **last**.
|
||||
- `rad_declare_encoder(RadEncoderDecl)` — a vision/audio tower, media preprocessed by the core.
|
||||
- `rad_declare_generator` / `rad_declare_audio_generator` / `rad_declare_video_generator` — image,
|
||||
audio and video generation programs; the engine sizes the step from the probe's `gen_*` fields
|
||||
before declare.
|
||||
- Optional format exports let a model state what it knows and the server learn nothing new:
|
||||
`rad_arch_chat_format` (a data description of reasoning/tool-call markup, replacing template
|
||||
analysis), `rad_arch_audio_format` (how a speech model hears: clip tokens, prompt, language,
|
||||
transcript start), `rad_arch_text_prepare` (rewrite request text into training form),
|
||||
`rad_arch_video_positions`/`rad_arch_video_pack` (video pass geometry and packing),
|
||||
`rad_arch_encode_geom` (encoder pass geometry). Each struct grows only at its end behind
|
||||
`struct_size`, so adding a field does not move the ABI.
|
||||
|
||||
**Media preprocessing is the core's**, not the plugin's: resizing, patching, placeholder expansion
|
||||
and positions in six processor families (Qwen-VL, Gemma 4, LFM2-VL, Unlimited-OCR, GLM-5.3, Pixtral
|
||||
for Nemotron), chosen by `vision_config.model_type` plus owning-model checks; audio through Gemma 4's
|
||||
extractor or Qwen3-ASR's Whisper one, chosen by `audio_config.model_type`. A vision model with another
|
||||
pipeline needs a core change.
|
||||
|
||||
## Step: issuing the graph
|
||||
|
||||
`RadBatch` is built once per step and shared by every block: `phase` (`PREFILL`/`DECODE`/`MIXED`),
|
||||
`n_tok`, `n_seq`, `token_ids`, `positions`, `cu_seqlens`, `seq_ids`, `lora_slot`, `n_out`/`out_ids`,
|
||||
`q_lens`/`ctx_lens`/`max_q_len`/`max_ctx_len`, the KV-group array, the speculative fields
|
||||
(`n_spec`, `draft_pass`, …).
|
||||
|
||||
```c
|
||||
static void step(RadCtx* c, const RadBatch* batch) {
|
||||
if (batch->draft_pass > 0) { m.ds.step(c, batch); return; } /* query pass */
|
||||
if (batch->draft_pass < 0) { m.ds.context(c, batch); return; } /* context pass */
|
||||
RAD_ISSUE(c, m.op_embed, praw(batch->token_ids, RAD_I32, batch->n_tok),
|
||||
RAD_W(m.w_tok), brows(m.b_x, batch->n_tok));
|
||||
for (auto& layer : m.layers) { layer.attn.step(c, batch); layer.mlp.step(c, batch); }
|
||||
/* ...heads; the lm_head only over batch->n_out rows... */
|
||||
}
|
||||
```
|
||||
|
||||
- `rad_issue(c, op, operands…, n)` / `RAD_ISSUE` (n = the batch's ranged extent) /
|
||||
`RAD_ISSUE_N` (an explicit row count). `rad_step_fail(c, what)` sets the status helpers return.
|
||||
- **A mixed step is the trap.** Continuous batching produces steps with a prefill chunk beside decode
|
||||
rows. A block that issues one path for the whole step sends its decode rows down the chunked path,
|
||||
which commits recurrent state directly and takes no `num_accepted`; a rejected draft cannot be
|
||||
rolled back, and the model stays fluent while drifting. Issue the recurrent path over
|
||||
`RadBatch::n_seq_decode` sequences and the chunked path over the rest (the core orders steps
|
||||
decode-first, so each half is contiguous; `cu_seqlens` holds absolute offsets, so pass
|
||||
`cu + n_seq_decode` with the token operands at their base).
|
||||
- A step issued on lane 1 outside the declared window fails by name. An op the core appends behind a
|
||||
declared one (runtime LoRA) counts at that op's position.
|
||||
- **Zero rows is not a call**: issuing an op over zero rows (a zero ranged extent, or all-empty
|
||||
sequences) returns `RAD_E_SHAPE` and writes nothing. The caller skips the issue, as the
|
||||
architectures' decode and prefill halves do.
|
||||
- `n_out` is 0 for a chunk that finishes no sequence — do not run the lm_head.
|
||||
|
||||
## Probe: what the plugin measured
|
||||
|
||||
`rad_arch_probe(meta, out)` runs before declare. Its fields are the engine's permissions:
|
||||
|
||||
| field | effect |
|
||||
|---|---|
|
||||
| `draft_depth`, `draft_depth_fixed` | the container's spec depth; a block drafter's trained block is the only depth it serves |
|
||||
| `shape_probe_ok` | the plugin tolerates a sizing declare, so the activation arena can be lent |
|
||||
| `enc_merge`, `audio_frames` | encoder pass granularity; longest audio clip |
|
||||
| `gen_rows`, `gen_patches`, `gen_audio`, `gen_ctx`, `gen_video` | generator sizing before declare |
|
||||
| `kv_quant_ok` | bits of the narrow KV layouts this model's kernels serve |
|
||||
| `chain_pays`, `tree_pays` | the plugin measured chained launches / draft trees faster |
|
||||
| `kv_stream_ok` | the architecture can stream a KV group (sparse, block-addressed, with `kv_resolve`) |
|
||||
| `control_vector_ok` | the plugin applies `RadBuildCtx::cvec` |
|
||||
| `tp_attn_rows_ok` | the plugin serves row-split prefill attention |
|
||||
| `state_f16_ok` | the plugin declares recurrent state at f16 and its kernels read it |
|
||||
|
||||
A flag whose probe bit is 0 is **refused at startup with the plugin named** (`--kv-cache-dtype
|
||||
fp8_v4`, `--control-vector`, `--tp-attn-split rows`, `--state-cache-dtype f16`, `--kv-resident`).
|
||||
That is the design: a deployment switch the plugin cannot honour is an error, never a silent
|
||||
substitution.
|
||||
|
||||
## Worked pattern: the smallest complete plugin
|
||||
|
||||
`arch/llama_fp8/llama_fp8.cpp` builds twice from one source — `llama_fp8` (quant `fp8_e4m3`) and
|
||||
`llama_dense_fp8` (quant `""`), differing only in the descriptor and the graph-dump text. It shows
|
||||
the whole shape:
|
||||
|
||||
1. `struct Layer { AttnFP8 attn; MlpFP8 mlp; }`, `struct Model` holding `Names`, `Geom`, buffer and
|
||||
op handles.
|
||||
2. `declare()`: name map entries first, then weights (with `rad_weight_encoding` where the head's
|
||||
format decides the kernel), buffers, KV group, the embedding and its all-reduce, then each layer's
|
||||
`declare()` from the shared blocks, the final norm, gather and lm_head, **the drafter last**, and
|
||||
`rad_note` lines explaining the shape.
|
||||
3. `step()`: draft passes first (sign selects query/context), then embedding (+ exact all-reduce
|
||||
when needed), each layer's `step()`, the final norm, and the head only over `n_out`.
|
||||
4. `probe()`: reads `draft.block_size` from the metadata and refuses any other depth for a block
|
||||
drafter.
|
||||
5. `RAD_ARCH_PLUGIN(...)` at the bottom.
|
||||
|
||||
Add a new family by copying a neighbour, replacing the name map and geometry, and composing the
|
||||
blocks you need; add a new *format* by cloning the plugin with another `(id, quant)` pair and the
|
||||
ops that format needs.
|
||||
|
||||
## Verifying a new plugin
|
||||
|
||||
```sh
|
||||
radiance --model m.rad --debug-graph # every op, its resolved kernel, and why
|
||||
radiance --model m.rad --debug-placement # the plan
|
||||
rad-convert $SRC -o m.rad --plan-only -v # a convert that plans will boot
|
||||
rad-kbench -m m.rad --kernels mylib,libr4d,libref --report k.md --bench
|
||||
rad-tune -m m.rad --tp 2 # only at the shapes you serve
|
||||
```
|
||||
|
||||
`tests/arch_test.cpp` drives a plugin's builder the way the core does, including lane-window
|
||||
liveness, so the pinned graphs and steps are the ones the core plans. A plugin that declares a weight
|
||||
form no kernel `layout` accepts is a declare-time error naming the weight, encoding and kernel — the
|
||||
same check `rad-convert` runs before writing.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Radiance Plugins]] — kernel libraries, encodings, the build path, ABI law.
|
||||
- [[Radiance Conversion]] — how the name map drives a convert, and `--set`/descriptor selection.
|
||||
- [[Radiance Model Families]] — the existing `(id, quant)` pairs and their recipes.
|
||||
- [[Radiance Tools]] — `rad-kbench`, `rad-tune`, `rad-info --plugins`.
|
||||
@@ -0,0 +1,354 @@
|
||||
---
|
||||
type: "howto"
|
||||
title: "Radiance Conversion — checkpoints into .rad containers"
|
||||
description: "Turning a checkpoint into a .rad container: rad-convert, the recipe language, quantisers, reuse and export."
|
||||
tags:
|
||||
- llm
|
||||
- radiance
|
||||
- quantization
|
||||
- howto
|
||||
timestamp: 2026-10-30
|
||||
---
|
||||
|
||||
# Radiance conversion
|
||||
|
||||
`rad-convert` turns a Hugging Face checkpoint into a `.rad` container, choosing each weight's
|
||||
encoding from a **recipe**. `rad-export` wraps it for finetunes (a merged checkpoint or a PEFT
|
||||
adapter) with the stock model's recipe. Neither opens a GPU: conversion is CPU-only and safe beside a
|
||||
running server.
|
||||
|
||||
Fetch checkpoints with `scripts/hfget.sh <org/repo> <dir>` (resumable; reads `.safetensors`,
|
||||
skips `.bin`/ONNX/GGUF when safetensors exist; `HF_TOKEN` for gated repos; writes a
|
||||
`.hfget-manifest` of content keys). `radiance --model <owner/name>` runs the same fetch itself.
|
||||
|
||||
## What a conversion does
|
||||
|
||||
1. Read `config.json` (or GGUF metadata), and the shard index if present.
|
||||
2. Load the architecture, kernel and quantiser plugins and run **declare**, as the engine does.
|
||||
3. Resolve every op and ask each resolved kernel's `layout` hook about the planned encoding. A
|
||||
recipe no kernel can serve is refused **here**, before any I/O.
|
||||
4. Write each logical weight once: the checkpoint's bytes where no rule matches, otherwise the
|
||||
planes the matching quantiser writes, a block of rows at a time.
|
||||
5. Measure every quantised weight against its source through the core's decoder and print the worst.
|
||||
6. Bake the tokeniser, chat template, metadata, recipe and any draft head.
|
||||
|
||||
```sh
|
||||
build/bin/rad-convert /srv/models/hf/Qwen/Qwen3.8-27B -o qwen3.8-27b-bf16.rad \
|
||||
--recipe data/recipes/q38-27b-bf16.recipe
|
||||
build/bin/rad-convert /srv/models/hf/Qwen/Qwen3.8-Flash-Next -o flashnext-bf16.rad --max-ctx 4096
|
||||
```
|
||||
|
||||
## Every `rad-convert` option
|
||||
|
||||
| option | meaning |
|
||||
|---|---|
|
||||
| `<input>` | a `.gguf`, a `.safetensors`, an `index.json`, or a checkpoint directory |
|
||||
| `-o, --out FILE` | the container to write |
|
||||
| `--recipe FILE` | one rule a line: `PATTERN QUANTISER key=value …`; repeatable; first match decides |
|
||||
| `--quant RULE` | `PATTERN=QUANTISER[:key=value,…]`, placed **ahead of** the recipe's rules; repeatable |
|
||||
| `--list-quantizers` | every loaded quantiser and the options it takes, then exit |
|
||||
| `--arch ID` | override the architecture id read from the checkpoint |
|
||||
| `--home DIR` | `$RADIANCE_HOME` (kernels, architectures, quantizers) |
|
||||
| `--kernels A:B` | kernel libraries to load, first wins; declare checks the recipe against them |
|
||||
| `--tokenizer F` | `tokenizer.json`; its declared chain is baked instead of reconstructed from a pre-tokenizer name. Required for a GGUF, without it the chain is a guess |
|
||||
| `--tokenizer2 F` | a second `tokenizer.json` baked as the second vocab (a generator reading its prompt through two tokenisers); keep `tokenizer_config.json` beside it |
|
||||
| `--add FILE` | merge one more `.safetensors`, tensor names kept; repeatable; a name in two files is refused |
|
||||
| `--config DIR` | read `config.json`, `generation_config.json`, `tokenizer.json` from DIR for a single-file checkpoint |
|
||||
| `--chat-template F` | a Jinja template baked in, overriding the checkpoint's (or its absence); trailing newlines stripped; without one `/v1/chat/completions` answers 501 |
|
||||
| `--imatrix F` | llama.cpp imatrix: expert-popularity profile, per-column importances, error weighting; required by `ggml` types `iq2_xxs`, `iq2_xs`, `iq1_s` |
|
||||
| `--set K=V` | a metadata key, seen by declare **and** written into the container; repeatable |
|
||||
| `--max-tok N` | the `max_tok` declare runs at (default 8192, or a generator's largest pass) |
|
||||
| `--max-ctx N` | the context declare runs at (default: the checkpoint's training context) |
|
||||
| `--max-spec N` | the `max_spec` declare runs at (default 4, or the block a drafter states); `0` leaves a draft head out |
|
||||
| `--draft-model DIR` | a drafter shipped as its own repository, merged under `draft.` |
|
||||
| `--kv-cache-dtype T` | `bf16` or `fp8`: the cache width declare runs at; the container is identical at either |
|
||||
| `--reuse FILE` | copy every weight an earlier container holds with the same encoding, quantiser, options and shape |
|
||||
| `--changed FILE` | with `--reuse`: checkpoint tensors (one name a line) that differ; every weight reading one is converted |
|
||||
| `--in-place` | with `--reuse`: extend that container; new weights go past its end, the header last; keeps `<file>.pre-append` |
|
||||
| `--lora FILE\|DIR` | merge a PEFT adapter into the checkpoint tensors before converting (`W + strength · scale · B·A`); repeatable, merged in order; each merged tensor joins `--changed` |
|
||||
| `--lora-strength F` | strength for every `--lora` (default 1) |
|
||||
| `--lora-map FILE` | JSON map for adapters whose module names differ: `{"strip":[prefix…],"modules":{"<pat>":[target…]}}`; one `*` matches a block index; several targets mean a fused module (rows split in order) |
|
||||
| `--plan-only` | declare, plan every encoding, print the plan, stop before writing |
|
||||
| `--report N` | print the N worst quantisation errors (default 20) |
|
||||
| `-v, --verbose` | per-weight detail |
|
||||
|
||||
`--reuse` matches a weight on its **encoding, quantiser, options and shape — not on its source
|
||||
bytes** — and carries whatever calibration made the old planes. Use it only when source and
|
||||
calibration are unchanged, or name the changed tensors with `--changed`. `--in-place` refuses to
|
||||
change a weight the container holds or to add an expert.
|
||||
|
||||
## The recipe language
|
||||
|
||||
```
|
||||
# PATTERN QUANTISER OPTIONS
|
||||
blk.*.ffn_*_exps.*.weight gptq codes=i4 group=128 scale=bf16 transform=fwht128 calib=$CALIB
|
||||
blk.*.ple_ngram.weight rtn codes=fp8_e4m3 block=*x* scale=bf16 rule=fixed scale_value=1.99317932128906e-4
|
||||
output.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16
|
||||
blk.*.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16
|
||||
```
|
||||
|
||||
- One rule a line: `PATTERN QUANTISER [key=value …]`. `#` starts a comment.
|
||||
- `PATTERN` is a glob over the **whole logical weight name**: `*` crosses dots, `?` is one character.
|
||||
- **The first matching rule decides.** No rule → the checkpoint's own encoding, which must be one
|
||||
the engine serves as is: `bf16`, `f16`, `f32`, or block-scaled FP8 (`X.weight` with
|
||||
`X.weight_scale_inv`). Anything else is named and refused.
|
||||
- `$NAME` / `${NAME}` in a value comes from the environment; an **unset variable is refused**, never
|
||||
expanded empty (so an empty `$CALIB` cannot silently produce an uncalibrated convert).
|
||||
- A quantiser may **decline** a weight (`RAD_E_UNSUPPORTED`), which keeps the checkpoint's encoding
|
||||
— `rtn` declines a rank-1 tensor, so a pattern that also matches a norm leaves the norm alone.
|
||||
- An option the quantiser does not declare is refused, by name, before anything is quantised.
|
||||
- Two plugins naming the same quantiser are refused, both named.
|
||||
|
||||
## Quantisers
|
||||
|
||||
| quantiser | what it does |
|
||||
|---|---|
|
||||
| `rtn` | round to nearest on a grid; each block's scale chosen by `rule` |
|
||||
| `gptq` | error feedback against the layer's input Hessian (`calib=DIR`, damped by `damp` = 0.01 of the mean diagonal by default); `act_order=1` quantises columns by descending input energy and stores the permutation; `cd=N` adds coordinate-descent sweeps |
|
||||
| `awq` | per-input-column scaling to a searched power, then a per-block clip search, both against the Hessian; column scales are the last scale level |
|
||||
| `autoround` | sign-gradient descent on a rounding offset per value and a range factor per block; `iters` steps; slow (a 27B dense model takes most of a working day at defaults; `iters=50` a quarter) |
|
||||
| `paroquant` | channel scaling plus `stages` Givens rotations learned against the Hessian, then AutoRound on the rotated weight; the `givens` transform |
|
||||
| `m2xfp4` | M2XFP's weight-side Sg-EM: MXFP4 plus one metadata byte per group of 32; its own `m2xfp4` scheme; `adaptive=0` pins the bias, `wide=1` widens it |
|
||||
| `ggml` | a llama.cpp format (`q4_0`, `q4_1`, `q5_0`, `q5_1`, `q8_0`, `q2_K`…`q6_K`, `iq4_nl`, `iq4_xs`, `iq2_xxs`, `iq2_xs`, `iq2_s`, `iq3_xxs`, `iq3_s`, `iq1_s`, `iq1_m`), by llama.cpp's own search, imatrix-weighted, codes byte for byte |
|
||||
| `cast` | a float weight in another float dtype (`dtype=` f32, bf16, f16, fp8_e4m3, fp8_e5m2), no scale |
|
||||
| `import` | rad-convert's own, not a plugin: write the numbers a quantised release ships, in the spelling the kernels read, instead of quantising again |
|
||||
|
||||
Every quantiser except `m2xfp4`, `ggml`, `cast` and `import` takes the grid options:
|
||||
|
||||
| option | values |
|
||||
|---|---|
|
||||
| `codes` | `i2`…`i8`, `i16`, `i32` symmetric; `u1`…`u8`, `u16` with `zero=`; `fp8_e4m3`, `fp8_e5m2`, `fp6_e2m3`, `fp6_e3m2`, `fp4_e2m1` |
|
||||
| `table` | a codebook the codes index: `nf4`, `iq4nl`, `w4nl`, `w5nl`, `binary`, `ternary`, or `;`-separated values |
|
||||
| `group=N` / `block=RxC` | where the scales sit; `*` is a whole extent |
|
||||
| `scale` | f32, bf16, f16, fp8_e4m3, e8m0; under `scale2` also an integer sub-scale `u4`…`u8`, `i6`, `i8` |
|
||||
| `scale2`, `group2` / `block2` | a second scale level the first is relative to (default the whole tensor) |
|
||||
| `scale2_value` | the second level fixed at this value, exact in `scale2`'s dtype |
|
||||
| `zero` | an integer zero point per block — an asymmetric grid |
|
||||
| `transform=fwhtN` | rotate each row by the Hadamard first |
|
||||
| `rule` | `absmax`, `mx` (OCP shared exponent), `fixed` (`scale_value=`), `search` (least squared error over candidates and refits, imatrix-weighted) |
|
||||
| `clamp` | a signed code's range: `full` or `sym` |
|
||||
|
||||
Named grids: MXFP4 = `codes=fp4_e2m1 group=32 scale=e8m0`; NVFP4 =
|
||||
`codes=fp4_e2m1 group=16 scale=fp8_e4m3 scale2=f32`; NF4 = `table=nf4 group=64`; a K-quant-like grid
|
||||
= `codes=i4 group=32 scale=u6 scale2=f16 group2=256`; a ternary BitNet weight = `table=ternary`.
|
||||
|
||||
**A container can hold only what some kernel reads.** rad-convert checks every planned encoding
|
||||
against the `layout` hooks of the kernels declare resolved and refuses one no kernel accepts. libr4d
|
||||
reads the plain floats; block-scaled fp8 and fp8 rows; int8 block rows plain or rotated; routed
|
||||
experts as fp8, int4 (`i4*bf16[1x128]/fwht128`) or the `w4nl`/`w5nl` codebooks under E4M3 scales;
|
||||
the dense w4, w8 and mxfp4 GEMM rows; and the 2-bit draft head. See [[Radiance Plugins]] for the
|
||||
encoding model.
|
||||
|
||||
## Importing a release someone else quantised
|
||||
|
||||
These forms are read weight by weight and written in the spelling the kernels read; a recipe rule
|
||||
re-quantises from the exact dequantisation where the family's kernels want another form.
|
||||
|
||||
| `quantization_config` | read as | notes |
|
||||
|---|---|---|
|
||||
| `quark` MXFP4 (fp4, group 32, e8m0) | `fp4_e2m1*e8m0[1x32]` | byte for byte; also servable as it stands |
|
||||
| `paroquant_mxfp4` | `fp4_e2m1*e8m0[1x32]/givens` | codes/exponents byte for byte plus learned rotations as `givens` tables |
|
||||
| `paroquant`, `int5-bitplane`, plain `paroquant` | `u5*f16[1x128]-u8[1x128]/givens`, `u4*…` | AWQ-packed nibbles re-laid as a code bit stream; f16 scales, integer zeros, rotations |
|
||||
| `compressed-tensors` `pack-quantized`, 4-bit `int`, group, act-order none/static/weight | `i4*<scale>[1xG]` | `weight_packed` made signed beside `weight_scale`; group/dynamic reorder is refused |
|
||||
| same with `symmetric:false` (AWQ) | `u4*<scale>[1xG]-u4[1xG]` | nibbles plus `weight_zero_point`, bit for bit |
|
||||
| `auto-round` `auto_round:auto_gptq`, `gptq` v1 | `i4*bf16[1x128]` or `i8*bf16[1x128]` | scales rounded to bf16; convert-time; `import` keeps shipped f16 |
|
||||
| `nvfp4-pack-quantized` | `fp4_e2m1*f32[1x16]` | `weight_scale / weight_global_scale`; **no kernel serves NVFP4**, so it always takes a rule |
|
||||
| `modelopt` NVFP4 / FP8 / MIXED_PRECISION | NVFP4 `fp4_e2m1*f32[1x16]`; FP8 `fp8_e4m3*f32[1x*]` | ModelOpt's second scale **multiplies**; `input_scale` is not read |
|
||||
| compressed-tensors all-FP8 groups | `fp8_e4m3*<scale>[1x*]` or `[RxC]` | nothing imported: stock FP8 with its scale; a per-tensor scale is written once a row |
|
||||
| GGUF `Q8_0`, `Q4_K`, `Q6_K` | `i8*f16[1x32]`, affine u4/u6…, `i6*i8…` | blocks split bit for bit; **not servable as they stand** |
|
||||
|
||||
The width is read **per tensor**, so a mixed release imports int4 experts and int8 projections
|
||||
together. What an import cannot read (an asymmetric GPTQ zero point, a group-reordering act-order
|
||||
index, a GPTQ group under 128, a missing/zero global scale, an unlisted format) is refused by name.
|
||||
The release's `quantization_config` names the tool, not a served form, so it is kept as
|
||||
`imported_quantization_config.*`.
|
||||
|
||||
`import` options: `codes=uN` makes a symmetric grid unsigned under a zero point of `2^(N-1)`;
|
||||
`zero=` the zero plane's dtype (`u4`, `u8`); `scale=` f16/bf16/f32, each shipped scale rounded to
|
||||
nearest; `block=RxC` repeats an FP8 scale shipped per row/tensor over that grid. It reads an imported
|
||||
integer grid or FP8 with a per-row/per-tensor scale and keeps a plain float weight as it is.
|
||||
|
||||
**NVFP4 to MXFP4 rounds twice** (e4m3 per 16 → power of two per 32), so it loses more than MXFP4 made
|
||||
from bf16. Prefer a calibrated MXFP4 container when one exists; the quality verdict is a KL gate, not
|
||||
a file size.
|
||||
|
||||
## The descriptor law (`--set`)
|
||||
|
||||
The architecture plugin is chosen by the exact `(arch id, quant descriptor)` pair. A convert-time
|
||||
choice that must also hold at serve time is stated once with `--set`, which hands it to declare and
|
||||
writes it into the container. Storage format is not such a choice (the encoding is in the container
|
||||
and the plugin reads it there), but the **serving plugin** is: a `quantization_config.*` key
|
||||
re-derives the descriptor.
|
||||
|
||||
```sh
|
||||
rad-convert <bf16 Qwen3.8-27B> -o q38-27b-fp8.rad \
|
||||
--recipe data/recipes/q38-27b-fp8block-df2.recipe \
|
||||
--set quantization_config.quant_method=fp8 --set quantization_config.fmt=e4m3 \
|
||||
--set "quantization_config.weight_block_size=128 128" \
|
||||
--set quantization_config.activation_scheme=dynamic
|
||||
```
|
||||
|
||||
A bf16 checkpoint whose recipe writes Qwen's block FP8 must therefore be declared *as that format*;
|
||||
otherwise the selection pair does not match any plugin and the convert (or the boot) is refused. The
|
||||
same law bites the other way when a release's own `quant_method` names a pair no plugin claims.
|
||||
|
||||
## Inspect and validate the result
|
||||
|
||||
```sh
|
||||
rad-info out.rad # architecture, source quant, recipe, sections
|
||||
rad-info --recipe out.rad # the expanded rules — did the intended one win?
|
||||
rad-info -v out.rad | head -40 # every tensor, its encoding, and what made it
|
||||
rad-kbench -m out.rad --home $RADIANCE_HOME --report kbench.md
|
||||
radiance --model out.rad --debug-graph # every op resolves
|
||||
```
|
||||
|
||||
The converter's own error report (the worst `--report N` weights) measures each quantised weight
|
||||
against its source through the core decoder. `rad-kbench` adds the artifact test: weight operands
|
||||
read from the container and passed through each kernel's `layout`/`relayout` hooks. Both are on
|
||||
[[Radiance Tools]].
|
||||
|
||||
## Calibration (GPTQ, AWQ, AutoRound, ParoQuant)
|
||||
|
||||
`calib=DIR` is a directory an engine run wrote with `RADIANCE_CALIB_DIR` set: each layer's **Gram
|
||||
matrices of its input**, in the domain the kernel multiplies it in. A block that characterises an
|
||||
activation declares an f32 `[K,K]` PERSIST buffer and a `gram_accum` op; at `RADIANCE_CALIB_TOKENS`
|
||||
(default 65536) the run writes the files.
|
||||
|
||||
Three-step bootstrap: convert with plain `rtn` → serve that container with `RADIANCE_CALIB_DIR` →
|
||||
push real prompts through prefill → convert again with the Hessians. The second moment is robust to
|
||||
the first container's small quantisation error. A weight with no file is quantised uncalibrated and
|
||||
the converter says so; `awq`, `autoround` and `paroquant` fall back to the imatrix diagonal when one
|
||||
is given.
|
||||
|
||||
- The accumulator must be **PERSIST**: a transient buffer may share bytes and be overwritten between
|
||||
the launches it sums.
|
||||
- Files are per rank; the consumer sums them. Under expert parallelism a rank's `down` input covers
|
||||
only its experts' tokens; `gate_up` input is the full token set on every rank, so the sum scales
|
||||
`H` by world size — GPTQ is invariant to a positive scale.
|
||||
- Under tensor parallelism a column-split linear sees only its rank's columns, so a tap may write
|
||||
`gram.r<rank>.<stem>.part.bin`; libquant places the parts on the diagonal in rank order, leaving
|
||||
cross-rank products zero. They must cover K and be in the model's own domain.
|
||||
- `--imatrix` supplies the expert-popularity profile the placement planner ranks units by, the
|
||||
per-column importances a quantiser uses, and the error report's weighting.
|
||||
|
||||
## Reuse, drafters, merging
|
||||
|
||||
```sh
|
||||
# re-quantise only what changed
|
||||
rad-convert $SRC -o new.rad --recipe new.recipe --reuse old.rad --changed changed.txt
|
||||
|
||||
# merge a drafter shipped as its own repo (DFlash2, DSpark, MTP assistant)
|
||||
rad-convert $SRC -o out.rad --draft-model /srv/models/z-lab/Qwen3.8-27B-DFlash2 \
|
||||
--recipe data/recipes/q38-27b-mxfp4-df2.recipe
|
||||
|
||||
# merge a PEFT adapter at convert time
|
||||
rad-convert $SRC -o tuned.rad --lora ~/run/adapter --lora-strength 1.0 \
|
||||
--recipe data/recipes/q38-27b-mxfp4-df2.recipe
|
||||
```
|
||||
|
||||
- A drafter's tensors go in under `draft.`, its config under `draft.` keys. It has no embedding and
|
||||
no `lm_head` of its own; it uses the target's.
|
||||
- `--lora-map` exists for adapters trained on names the checkpoint does not use; a fused target
|
||||
(`qkv`) takes several targets and splits the adapter's rows in order.
|
||||
- `--add` + `--config` + `--tokenizer2` assemble a ComfyUI-style release of separate single files
|
||||
(Anima: diffusion model, text encoder, VAE). A name in two files is refused.
|
||||
|
||||
## `rad-export` — finetunes without hand-writing the command
|
||||
|
||||
```sh
|
||||
rad-export ~/outputs/my-run/checkpoint-400 --dry-run # what it would run
|
||||
rad-export ~/outputs/my-run/checkpoint-400 # adapter -> <name>.rad
|
||||
rad-export ~/exports/merged -o ~/models/mine.rad --recipe q38-27b-fp8block-df2
|
||||
```
|
||||
|
||||
| flag | default | meaning |
|
||||
|---|---|---|
|
||||
| `SRC` | required | a merged 16-bit checkpoint (`config.json` + `*.safetensors`) or a PEFT adapter (`adapter_config.json` + `adapter_model.safetensors`) |
|
||||
| `-o, --out FILE` | SRC's folder name + `.rad`, beside SRC or in `out_dir` | the container |
|
||||
| `--recipe NAME\|FILE\|auto` | `auto` | a recipe name, a file, or the one recipe whose header names the base model, takes a bf16 checkpoint and needs no calibration |
|
||||
| `--base DIR` | the adapter's `base_model_name_or_path`, from disk or the HF cache, else downloaded | the base a LoRA merges into |
|
||||
| `--draft DIR\|ORG/NAME` | the config, else the Hub repo the recipe's usage line names | the drafter when the recipe merges one; a usage line with a placeholder asks for `--draft` |
|
||||
| `--image IMAGE` | `$RADIANCE_IMAGE`, the config's, then `stilldeadcode/radiance` | run `rad-convert` in this image (docker, CPU-only, no network, **not pulled**) |
|
||||
| `--rad-convert PATH` | — | run this binary instead of docker; recipes come from its `share/radiance` |
|
||||
| `--memory M` | — | docker memory limit for the convert |
|
||||
| `--dry-run` | — | resolve recipe, drafter and paths; print the `rad-convert` line; write nothing |
|
||||
| `--keep-merged` | — | keep the merged checkpoint beside OUT |
|
||||
| `--config FILE` | `~/.config/rad-export/config.json` | the config below |
|
||||
| `--list-recipes [--json]` | — | list the recipes the image/installed tool carries |
|
||||
|
||||
- Input: a merged 16-bit checkpoint (`config.json` + `*.safetensors`) or a PEFT LoRA adapter
|
||||
(`adapter_config.json` + `adapter_model.safetensors`).
|
||||
- An adapter is merged **exactly as PEFT's `merge_and_unload`** computes it on CPU: `scale` =
|
||||
alpha/r (alpha/√r with rsLoRA), `rank_pattern`/`alpha_pattern` honoured, delta in fp32 rounded to
|
||||
the adapter's dtype. It streams tensor by tensor into 1 GiB shards, so a 27B merge needs a few GiB
|
||||
of RAM, not the model twice. Plain linear LoRA only: embedding LoRA, trained biases, DoRA, saved
|
||||
modules, convolution LoRA, a rank other than the config's, and a missing target are refused before
|
||||
anything is written. The adapter's tokeniser/template replace the base's where shipped.
|
||||
- The recipe comes from the image (`/opt/radiance/share/radiance/recipes`) or the installed
|
||||
`rad-convert`'s share; `auto` matches the base model's name in recipe headers and asks when several
|
||||
fit. A recipe needing calibration is listed but never picked by `auto` — a finetune's Hessians are
|
||||
not the stock model's.
|
||||
- It runs `rad-convert` on the CPU, under Docker as the caller with no network, and writes the log to
|
||||
`OUT.log`.
|
||||
- `~/.config/rad-export/config.json` pins what auto cannot know: `image`, `out_dir`,
|
||||
`recipes{model:recipe}`, `drafts{recipe:dir}`, `bases{repo:dir}`.
|
||||
- It is Python 3 with no dependencies; merging needs `torch`/`safetensors`, fetching a missing base or
|
||||
drafter needs `huggingface_hub`. The image carries no Python; extract the script with
|
||||
`docker run --rm --entrypoint busybox stilldeadcode/radiance cat /opt/radiance/bin/rad-export`.
|
||||
|
||||
## Scoring a container (the KL mode)
|
||||
|
||||
```sh
|
||||
# once, on the bf16 container
|
||||
radiance --model bf16.rad --kld-record /srv/kld/q38 --kld-corpus corpus.jsonl
|
||||
# each candidate, same flags otherwise
|
||||
radiance --model quant.rad --kld-ref /srv/kld/q38 --kld-out report.json
|
||||
```
|
||||
|
||||
- The corpus is JSONL, one `{"prompt", "score_from"?, "source"?}` a line; only `prompt` is required.
|
||||
Positions `[score_from, n-1)` are scored; `source` groups the report. Keep the scoring corpus
|
||||
disjoint from the calibration corpus.
|
||||
- `--kld-record` writes the reference's log-probabilities (f16 over the vocabulary, the 16 most
|
||||
likely tokens exactly, the token ids). DIR is never overwritten and is read only once `kld.json`
|
||||
exists, which is written last.
|
||||
- `--kld-ref` scores this model's f32 logits against stored rows: mean, median, 99/99.9/99.99th
|
||||
percentile KL, top-1 agreement, and both models' perplexity. `--kld-out` writes the report; `FILE.rows`
|
||||
is `[positions, 4]` f32 (KL, both NLLs, argmax agreement) for position-by-position comparison.
|
||||
- The mode turns speculation and the prefix cache off, turns the step pipeline off, and refuses an
|
||||
explicit spec depth above 0. The logits buffer holds a row per token of a step, so a large
|
||||
`--max-num-batched-tokens` costs pinned host memory as well as VRAM.
|
||||
- `--kld-lora NAME` scores every document through a served adapter, against a reference recorded
|
||||
from the merged checkpoint; `--kld-lora-sources PREFIX` runs only matching documents through it,
|
||||
the rest as the base, in the same steps — each document's rows should be byte-identical to the
|
||||
all-adapter and all-base runs.
|
||||
|
||||
The gate the fleet uses (mean KL ≤ 0.03, top-1 ≥ 95 %) is on [[frank-archon]] and
|
||||
[[Quantization Production]]; the mode here is only the instrument.
|
||||
|
||||
## Convert laws paid for in full
|
||||
|
||||
- **`--plan-only` first.** A plan that resolves every op will boot; it costs seconds against an hour
|
||||
of I/O on a large checkpoint.
|
||||
- **Declare runs at the step and context bounds, and some graphs depend on them.** Qwen4-Exp sizes
|
||||
its QSA buffers by `--max-ctx` and refuses a `--max-tok`/context pair whose score plane would pass
|
||||
2 GiB; the defaults (8192 tokens, the checkpoint's training context) fit most models, and a
|
||||
generator's largest pass may override `--max-tok`.
|
||||
- **Scale names follow the tensor.** `scale_name()` appends `_scale_inv` to the *tensor* name, so a
|
||||
stacked `…experts.gate_up_proj` keeps its scales at `…gate_up_proj_scale_inv`.
|
||||
- **File shadowing: first wins.** The reader opens index-named files sorted and emplaces every tensor
|
||||
each file physically holds, first-wins. A merged PLE shard must sort first
|
||||
(`00-ple.merged.safetensors`) or a raw shard wins the name.
|
||||
- **Never write through a staging symlink.** A checkpoint may symlink its raw index; `open(path,"w")`
|
||||
writes through it and corrupts the checkpoint. Write a temp file and `os.replace`.
|
||||
- **One encoding per expert table.** A whole expert kept bf16 is the only thing legal beside a
|
||||
table; a split tier (down=fp8, gate_up=w4) is unservable.
|
||||
- **The descriptor pair has no fallback** (above), and **convert is card-less**: GPTQ solves on CPU
|
||||
BLAS, so the GPU is irrelevant to production and the card only decides what can be decoded.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Radiance Model Families]] — the (arch id, quant descriptor) table and the command per model.
|
||||
- [[Radiance Plugins]] — quantiser and kernel-plugin contracts behind these options.
|
||||
- [[Radiance Tools]] — `rad-info`, `rad-kbench`, `rad-tune`.
|
||||
- [[Quantization Production]] — the fleet's calibration/KLD campaign and its gates.
|
||||
- [[frank-archon]] — the deployment whose recipes and convert laws these are.
|
||||
@@ -0,0 +1,140 @@
|
||||
---
|
||||
type: "reference"
|
||||
title: "Radiance Engine — the inference server behind frank-archon"
|
||||
description: "The radiance inference engine: its shape, plugin model, container format, and how the reference pages fit together."
|
||||
tags:
|
||||
- llm
|
||||
- radiance
|
||||
- reference
|
||||
timestamp: 2026-10-30
|
||||
---
|
||||
|
||||
# Radiance Engine — shape, plugin model, container
|
||||
|
||||
Radiance is StillDeadcode's inference server for AMD GPUs, written in C++20 and HIP: language
|
||||
models, plus image, video, speech and music generation and embeddings. One HTTP front end speaks
|
||||
OpenAI's chat, completions, Responses, images, audio and video APIs, Anthropic's Messages API and
|
||||
llama-server's native one. Nothing in the serving path is Python.
|
||||
|
||||
The fleet's only radiance endpoint is [[frank-archon]] (2× R9700, TP=2, Dankserver). **This page
|
||||
and its siblings own the engine reference; the live deployment shape, its Eras and its measured
|
||||
laws stay on the endpoint page.** The numbers here describe upstream `main` as read 2026-10-30
|
||||
(release 1.14.0, plugin ABI 31). The fleet pin is older (1.8.1): a flag below that post-dates the pin
|
||||
is a boot refusal, never a silent no-op, and the endpoint page records the CLI families that pin
|
||||
itself retired relative to its own rollback target.
|
||||
|
||||
Source: `https://codeberg.org/StillDeadcode/radiance` at `8da043c`. Engine docs live in `docs/`
|
||||
(`GUIDE.md` for serving, `TOOLS.md` for tools, `PLUGIN.md` for plugins, `ARCHITECTURES.md` per
|
||||
family, `OPS.md` for the op vocabulary, `spec.md` for the design).
|
||||
|
||||
## The four moving parts
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[Checkpoint<br/>safetensors / GGUF] -->|rad-convert + recipe| B[.rad container]
|
||||
B --> C[radiance engine]
|
||||
C --> D[OpenAI / Anthropic /<br/>llama-server wire APIs]
|
||||
P1[kernel library .so<br/>libr4d, libavx, libref] --> C
|
||||
P2[architecture .so<br/>one per arch id × quant descriptor] --> C
|
||||
P3[quantiser .so<br/>libquant] -->|rad-convert| A
|
||||
```
|
||||
|
||||
- **Engine** — `core/`: declare phase, placement/memory planner, scheduler, KV manager, sampler,
|
||||
tokeniser, HTTP server, media preprocessing.
|
||||
- **Kernel libraries** — implement ops for hardware. `libr4d` targets RDNA4 (`gfx1200`,
|
||||
`gfx1201`); `libavx` serves the host backend (AVX-512/AVX2) with no GPU at all; `libref` is the
|
||||
slow reference implementation every other library is checked against.
|
||||
- **Architecture plugins** — declare one model family at one weight format: its weights, buffers,
|
||||
KV groups and the ops it issues each step. The engine picks one by an **exact (architecture id,
|
||||
quantisation descriptor) pair, with no fallback**.
|
||||
- **Quantiser plugins** — `libquant`'s `rtn`, `gptq`, `awq`, `autoround`, `paroquant`, `m2xfp4`,
|
||||
`ggml`, `cast`, `import`; they write a weight's numbers into an encoding's planes.
|
||||
- **Tools** — `rad-info`, `rad-convert`, `rad-export`, `rad-kbench`, `rad-tune`, `rad-schemas`,
|
||||
`rad-tokcheck`, plus `rad-chatparse`, `rad-avx-check`, `r4d_selftest` in the build tree.
|
||||
|
||||
## Two phases, one static graph
|
||||
|
||||
`rad_arch_declare` runs once per rank and names every weight, buffer, KV group and op, in order.
|
||||
The core resolves every op against the kernel hierarchy per band, per domain, and completes the
|
||||
whole plan before a byte of weights is read. `rad_arch_step` then only *issues* handles obtained at
|
||||
declare; it may not allocate and may not synchronise. A model whose declare succeeds is fully known
|
||||
statically, which is what makes `--debug-graph`, the buffer liveness plan and `rad-convert`'s
|
||||
plan-only check possible.
|
||||
|
||||
Consequences a plugin author must respect:
|
||||
|
||||
- An op that never resolves is `RAD_NULL_HANDLE` and only fails if issued. A fusion is a *selection*:
|
||||
ask for the fused op, emit the chain when `rad_op_resolved` is false.
|
||||
- Declaration order is the buffer planner's liveness. A drafter must be declared after the trunk
|
||||
it runs behind, or trunk and drafter buffers silently share bytes.
|
||||
- Every op in the graph must have a row in a loaded kernel library. `libref` implements the
|
||||
conventional vocabulary in `docs/OPS.md`; device-only ops (most fusions) exist only in `libr4d`.
|
||||
|
||||
## The `.rad` container (RAD2)
|
||||
|
||||
A container is one file holding everything the server needs: weights as **canonical encoding
|
||||
planes**, the tokeniser (and optionally a second vocab), the chat template, metadata, the recipe it
|
||||
was made with, and any draft head. It names no kernel library.
|
||||
|
||||
- Planes are kernel-independent: codes, scales, zero points, codebooks — row-major, bit-packed
|
||||
sub-byte where needed. A kernel rearranges them into its own layout at load (`layout`/`relayout`
|
||||
hooks), losslessly, so a kernel library can change layouts without reconverting the model.
|
||||
- Encodings are data, not an enum: `RadEncoding` names the scheme (`plain`, `affine`, or a
|
||||
quantiser's own), the planes by role, the scale grouping, and a transform (`fwhtN`, `perm`,
|
||||
`givens`). That is how one file can hold block-FP8, int4, MXFP4, NVFP4-imported, ParoQuant
|
||||
rotations and llama.cpp grids.
|
||||
- `rad-info` is the reader: sections (strings, metadata, directory, planes, encodings, vocab,
|
||||
profile, data), `--meta`, `--vocab`, `--recipe`, `--plugins`, and `-v` for every tensor with its
|
||||
encoding and what made it.
|
||||
|
||||
See [[Radiance Conversion]] for producing one and [[Radiance Tools]] for inspecting it.
|
||||
|
||||
## Selection key and version law
|
||||
|
||||
- A checkpoint's architecture id is its `config.json` `model_type` without underscores; `rad-convert
|
||||
--arch ID` overrides it. `rad_arch_quant` supplies the quantisation descriptor (empty for an
|
||||
unquantised checkpoint).
|
||||
- **No fallback.** A raw official FP8 checkpoint whose `quantization_config.quant_method` is `fp8`
|
||||
is not the pair the `qwen4exp_fp8` plugin claims (`(qwen4exp, "")`), so both serve and convert
|
||||
refuse it until the descriptor is changed with `--set` or a staged `config.json`. The deployment
|
||||
lesson is on [[frank-archon]]; the mechanism is on [[Radiance Conversion]].
|
||||
- `RAD_ABI_VERSION` covers every header a plugin compiles against. A plugin reporting a different
|
||||
number is refused at load, with both numbers printed. It moves on any shape change, not on a new
|
||||
optional export; rebuild when it changes. Release version and ABI are independent: `/version`
|
||||
reports the release, `rad-info --plugins` reports what loaded.
|
||||
|
||||
## Environment
|
||||
|
||||
| variable | meaning |
|
||||
|---|---|
|
||||
| `RADIANCE_HOME` | plugin search path: `:`-separated homes, each holding `kernels/`, `architectures/`, `quantizers/`; first wins, same-name files shadow later homes. Compiled-in default is `<prefix>/share/radiance` |
|
||||
| `RADIANCE_KERNELS` | default for `--kernels`: ordered kernel libraries, first wins; unnamed libraries are not loaded |
|
||||
| `RADIANCE_TUNE` | pin a kernel's tuned axes for a measurement (a measurement aid, not a deployment setting) |
|
||||
| `RADIANCE_CALIB_DIR` | a serve run records per-layer input Gram matrices here (the GPTQ calibration tap) |
|
||||
| `RADIANCE_CALIB_TOKENS` | tokens after which the tap stops and writes (default 65536) |
|
||||
| `RADIANCE_LOG`, `RADIANCE_LOG_STEPS` | log level; per-step token counts |
|
||||
| `RADIANCE_DEBUG_RESID` | residual-stream probe per (layer, site) |
|
||||
| `RADIANCE_DFLASH_Q_TEMP` | temperature of a DFlash2 coupled walk's candidate softmax |
|
||||
|
||||
The full list of debug knobs (`RADIANCE_DEBUG_*`, `RADIANCE_DUMP_*`, `RADIANCE_OP_ORACLE`, …) is in
|
||||
the source's `getenv` calls; none is a deployment setting.
|
||||
|
||||
## Where the pages are
|
||||
|
||||
| Page | Owns |
|
||||
|---|---|
|
||||
| [[Radiance Serve Options]] | every serve flag, grouped, with defaults and the law behind each knob |
|
||||
| [[Radiance HTTP API]] | endpoints, request fields, streaming, tool calls, refusal rules |
|
||||
| [[Radiance Conversion]] | `rad-convert`, the recipe language, quantisers, imports, reuse, `rad-export`, KL scoring |
|
||||
| [[Radiance Model Families]] | which plugin claims which model, and the conversion command per family |
|
||||
| [[Radiance Plugins]] | kernel and quantiser plugins: ABI, rows, hooks, encodings, build and checks |
|
||||
| [[Radiance Architecture Plugins]] | declaring a model: weights, buffers, KV groups, ops, step, probe |
|
||||
| [[Radiance Tools]] | `rad-info`, `rad-kbench`, `rad-tune`, `rad-schemas`, `rad-tokcheck`, harnesses |
|
||||
| [[Radiance Operations]] | the container contract, host prerequisites, readiness, ledgers, restarts |
|
||||
| [[frank-archon]] | the fleet's live radiance endpoint: shape, Eras, rollback, measured laws |
|
||||
|
||||
## Sources
|
||||
|
||||
- `README.md`, `docs/GUIDE.md` §1–§4, `docs/TOOLS.md`, `docs/PLUGIN.md`, `docs/DOCKER.md`,
|
||||
`docs/ARCHITECTURES.md`, `docs/OPS.md`, `spec.md` §2–§4, `abi/*.h`, `core/config.cpp`,
|
||||
`core/rad_core.h` at commit `8da043c` (codeberg.org/StillDeadcode/radiance).
|
||||
@@ -0,0 +1,198 @@
|
||||
---
|
||||
type: "reference"
|
||||
title: "Radiance HTTP API — endpoints, fields, refusals"
|
||||
description: "The HTTP surface radiance serves: endpoints, request fields, streaming, tools, and refusal rules."
|
||||
tags:
|
||||
- llm
|
||||
- radiance
|
||||
- api
|
||||
- reference
|
||||
timestamp: 2026-10-30
|
||||
---
|
||||
|
||||
# Radiance HTTP API
|
||||
|
||||
One server serves one model. It answers OpenAI's chat/completions/Responses/images/audio/video APIs,
|
||||
Anthropic's Messages API and llama-server's native one, so their clients and SDKs work unchanged.
|
||||
`model` in a request is accepted and **not enforced**.
|
||||
|
||||
## Endpoints
|
||||
|
||||
| route | what it serves |
|
||||
|---|---|
|
||||
| `POST /v1/chat/completions` | chat: tools, reasoning, streaming, grammars |
|
||||
| `POST /v1/completions` | raw text completion |
|
||||
| `POST /v1/embeddings` | an embedding head, else 501 |
|
||||
| `POST /v1/images/generations`, `/v1/images/edits` | OpenAI images API; needs an image model, else 501 |
|
||||
| `POST /v1/systemone` | LiquidAI d1 decisions: named yes/no, choice and score questions as probabilities; needs a d1 container, else 501 |
|
||||
| `POST /v1/audio/transcriptions` | OpenAI speech-to-text; needs a transcribing plugin, else 501 |
|
||||
| `GET /v1/realtime` (WebSocket) | streaming transcription session |
|
||||
| `POST /v1/audio/speech` | speech (Breeze-TTS-2, drama-3) or a song (MiniMax-Music3), whole or streamed |
|
||||
| `GET /v1/audio/speech/stream` (WebSocket) | text streamed in, each sentence's audio out |
|
||||
| `POST /v1/videos`; `GET /v1/videos[/:id[/content]]`; `DELETE /v1/videos/:id` | video jobs made in the background, downloaded as MP4 |
|
||||
| `POST /v1/messages`, `/v1/messages/count_tokens` | Anthropic Messages, for Claude Code and the Anthropic SDKs |
|
||||
| `POST /v1/responses`, `/v1/responses/input_tokens` | OpenAI Responses, for Codex and SDK clients |
|
||||
| `POST /completion`, `/completions`, `/infill` | llama-server native completion and fill-in-the-middle |
|
||||
| `GET /v1/models`, `/v1/models/:id` | what is loaded, with `max_model_len`, `root`, `capabilities` |
|
||||
| `GET /health`, `/v1/health`, `/ping`, `POST /ping` | liveness |
|
||||
| `GET /load`, `/version`, `/props`, `/server_info`, `/diagnostics` | load; release version; llama-server props; full configuration; the startup report as JSON (`/diagnostics` is same-machine-only without an API key) |
|
||||
| `POST /apply-template`, `/v1/chat/completions/input_tokens` | the rendered prompt; its token count |
|
||||
| `GET /metrics` | Prometheus |
|
||||
| `POST /tokenize`, `/detokenize`; `GET /get_tokenizer_info` | vLLM's shape and llama-server's at once |
|
||||
| `POST /reset_prefix_cache`; `DELETE /sessions/<label>`, `/sessions?id=<id>` | drop every cached prefix; forget a stored conversation |
|
||||
| `GET /`, `/dashboard`; `GET /stats`, `/graph`, `/sessions` | the dashboard and the JSON it is drawn from |
|
||||
|
||||
Unversioned shorthands exist for chat, chat input tokens, embeddings, images, audio transcriptions,
|
||||
audio speech, models, responses and completions, for a client given the bare host as base URL.
|
||||
`/completions` **without** `/v1` is llama-server's native completion. A path nothing serves is a JSON
|
||||
404.
|
||||
|
||||
A text prompt to `/v1/completions`, `/v1/embeddings` or `/tokenize` reads control tokens as tokens
|
||||
(as vLLM and llama.cpp do), so a client rendering its own template can send `<|im_start|>user\n…`.
|
||||
The model's declared BOS is prepended unless the prompt already opens with it.
|
||||
|
||||
## Chat
|
||||
|
||||
```sh
|
||||
curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
|
||||
-d '{"model":"m","messages":[{"role":"user","content":"hi"}],"max_tokens":200,"temperature":0.7}'
|
||||
```
|
||||
|
||||
- **No chat template → 501** on this route (the container carries one; `--override-chat-template`
|
||||
replaces it). Use `/v1/completions` instead.
|
||||
- Thinking off has three spellings: `chat_template_kwargs.enable_thinking=false`,
|
||||
top-level `enable_thinking=false`, or `reasoning_effort:"none"`.
|
||||
- Reasoning returns in `choices[].message.reasoning_content`, separate from `content`.
|
||||
**Reasoning counts against `max_tokens`**; a reply that runs out inside it has empty `content` and
|
||||
`finish_reason:"length"`. `usage.completion_tokens_details.reasoning_tokens` tells that case from a
|
||||
model that answered nothing.
|
||||
- `thinking_token_budget` (aliases `thinking_budget`, `thinking_budget_tokens`,
|
||||
`reasoning_budget_tokens`) closes the reasoning block at N tokens and forces the close, so it holds
|
||||
under speculation and at any temperature. It needs a one-token end marker; otherwise the response
|
||||
has no `completion_tokens_details` and a budget is a 400.
|
||||
- `continue_final_message` continues a conversation ending in an assistant message instead of
|
||||
answering it; `add_generation_prompt` opens a new turn.
|
||||
- Tool calls: `tools`, `tool_choice` (`auto`, `none`, `required`, named function),
|
||||
`parallel_tool_calls`. `required`/named hold the whole reply to calls after the reasoning block and
|
||||
cannot combine with `response_format` or `grammar`.
|
||||
- Structured output: `response_format` (`json_object` or `json_schema`), llama-server's `json_schema`
|
||||
shorthand, or a raw GBNF `grammar`. A chat template that drops the grammar makes the request a
|
||||
refusal.
|
||||
|
||||
## Streaming
|
||||
|
||||
```sh
|
||||
curl -N localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
|
||||
-d '{"model":"m","messages":[{"role":"user","content":"Count to ten."}],
|
||||
"max_tokens":100,"stream":true,"stream_options":{"include_usage":true}}'
|
||||
```
|
||||
|
||||
Server-sent events in OpenAI's delta format, terminated by `data: [DONE]`. **Nothing is sent while
|
||||
the prompt is read**: the first event (the `role: assistant` chunk) goes out with the first generated
|
||||
token, so a client timing its first event measures time-to-first-token. `return_progress` adds
|
||||
llama-server's `prompt_progress` chunks during prefill; `timings_per_token` adds `timings` to every
|
||||
chunk.
|
||||
|
||||
## Completions, embeddings, images, audio, video, tokens
|
||||
|
||||
- **Completions** — `/v1/completions` takes `prompt` (string or token ids), `echo`, `logprobs`,
|
||||
`preserved_tokens`. `/completion` is llama-server's native route; `/infill` is fill-in-the-middle.
|
||||
- **Embeddings** — `input` is text, a batch, or token ids; `encoding_format` `float`/`base64`;
|
||||
`dimensions` truncates to the first N and renormalises; `prompt_name` prepends the model's named
|
||||
prompt. Pooling is the model's; the engine does not renormalise.
|
||||
- **Images** — `/v1/images/generations` returns `b64_json`; `size` defaults to 2048×2048. Edits take
|
||||
one picture or up to ten (`image[]`), multipart as OpenAI's SDKs send them, with an optional mask
|
||||
(repaint only where transparent). `partial_images` streams previews as SSE.
|
||||
- **Speech to text** — `/v1/audio/transcriptions`; `/v1/realtime` streams audio in and words back as
|
||||
they are spoken. The model's prompt shape comes from its plugin's audio format, not from the server.
|
||||
- **Speech and music** — `/v1/audio/speech` (whole or streamed) with `instructions` describing a voice
|
||||
(Breeze-TTS-2) or speakers written in the input (`<|speaker:1|>`, `[laughs]` for drama-3); a music
|
||||
model takes the lyrics as `input` and the description as `instructions`. `/v1/audio/speech/stream`
|
||||
streams text in and audio out as each sentence completes.
|
||||
- **Video** — `/v1/videos` is a job: a video with its stereo soundtrack from a prompt, 5–14 s, up to
|
||||
1344×768, made in the background; `GET /v1/videos/:id/content` downloads the MP4.
|
||||
- **Tokenising** — `POST /tokenize` accepts vLLM's shape (`prompt` or `messages` in, `prompt` out) and
|
||||
llama-server's (`content` in/out, `with_pieces`). `POST /detokenize` reverses it.
|
||||
`GET /get_tokenizer_info` reports the tokeniser configuration.
|
||||
|
||||
## Request fields
|
||||
|
||||
Accepted fields, in OpenAI's, vLLM's and llama-server's spellings:
|
||||
|
||||
| field | notes |
|
||||
|---|---|
|
||||
| `model` | accepted, not enforced |
|
||||
| `messages` / `prompt` / `input` | chat / completion / embedding |
|
||||
| `max_tokens`, `max_completion_tokens`, `n_predict` | default is what the context leaves; `-1` is llama-server's "no limit"; clamped to context and to `--max-tokens-cap` |
|
||||
| `temperature`, `top_p`, `top_k`, `min_p`, `typical_p` | sampler |
|
||||
| `presence_penalty`, `frequency_penalty`, `repetition_penalty`/`repeat_penalty`, `repeat_last_n` | penalties |
|
||||
| `xtc_probability`, `xtc_threshold`, `dry_*` | XTC and DRY |
|
||||
| `seed` | absent means "pick one", not 0 |
|
||||
| `n` | 1..`--max-n`; prefix cache makes the fan-out nearly free after the first |
|
||||
| `stop` | string or array, bounded by `--max-stop-strings`/`--max-stop-bytes` |
|
||||
| `priority` | scheduler hint |
|
||||
| `ignore_eos` | generate to `max_tokens` whatever the model emits |
|
||||
| `stream`, `stream_options.include_usage`, `timings_per_token`, `return_progress` | streaming controls |
|
||||
| `samplers` | llama-server's order; this engine's chain is penalties, DRY, top-k, top-p, min-p, typical, XTC, with temperature first or last; any other list is refused by name |
|
||||
| `session_id` | labels the conversation on `/sessions`, dashboard and `DELETE` |
|
||||
| `control_vector` | `true` applies the deployment's vector to this request; absent takes the deployment default; a deployment with no vector refuses `true` |
|
||||
| `logprobs`, `top_logprobs`, `n_probs` | up to 20 alternatives; not with `echo` |
|
||||
| `tools`, `tool_choice`, `parallel_tool_calls`, legacy `functions`/`function_call` | chat only |
|
||||
| `add_generation_prompt`, `chat_template_kwargs`, `continue_final_message` | chat only |
|
||||
| `reasoning_format`, `reasoning_effort`, `enable_thinking`, `thinking_token_budget`, `preserve_thinking` | chat only; levels are the template's (`capabilities.reasoning_efforts`); `none`/`minimal` turn thinking off |
|
||||
| `response_format`, `json_schema`, `grammar` | structured output; `grammar` and `response_format` are mutually exclusive |
|
||||
| `echo`, `preserved_tokens` | `/v1/completions` only |
|
||||
| `encoding_format`, `dimensions`, `prompt_name`, `image_tokens` | `/v1/embeddings` only |
|
||||
| `images_config` | image models only: a named processor mode or an object restating served sizes |
|
||||
|
||||
## What is refused, ignored, and clamped
|
||||
|
||||
The contract: **a 200 means every field that could change the answer was honoured.** Fields fall
|
||||
into four classes.
|
||||
|
||||
- **Inert** — accepted and ignored because they cannot change the answer: OpenAI/OpenRouter
|
||||
bookkeeping (`user`, `metadata`, `store`, `service_tier`, `prompt_cache_key`, …) and llama-server's
|
||||
own speed/slot knobs (`cache_prompt`, `id_slot`, `n_cache_reuse`, `backend_sampling`,
|
||||
`speculative.*`, …). The prefix cache and speculation here are the server's and never change output.
|
||||
- **Guarded** — known, unimplemented, and able to change the answer: accepted at their off value,
|
||||
refused by name otherwise. `logit_bias` (empty), `mirostat` (0), `top_n_sigma` (≤0),
|
||||
`dynatemp_range` (≤0), `min_keep` (0), `post_sampling_probs` (false), `best_of` (≤1), `suffix`
|
||||
(empty), `lora` (empty — an adapter is chosen by `model`), `modalities`/`audio` (`["text"]`/null),
|
||||
`n_indent` (0), `t_max_predict_ms` (≤0), `return_tokens` (false), `grammar_lazy`,
|
||||
`grammar_triggers`, `preserved_tokens` (empty on chat).
|
||||
- **Ignored** — anything else unknown, logged at debug. An unknown field is far more often a client's
|
||||
extension than a misunderstood request.
|
||||
- **`max_tokens` is the exception**: if prompt + `max_tokens` exceeds the context, it is **clamped**
|
||||
and served, ending `finish_reason:"length"`, because refusing would produce no tokens instead of
|
||||
fewer. A prompt that does not fit on its own is a 400 `context_length_exceeded` naming
|
||||
`n_prompt_tokens`, `n_ctx`, and how many tokens must go.
|
||||
|
||||
## Anthropic, Responses, llama-server
|
||||
|
||||
- `/v1/messages` is translated into a chat request, so chat defaults, refusals and limits apply.
|
||||
Streamed (`message_start`, content blocks, `message_delta`, `message_stop`) or not:
|
||||
`ANTHROPIC_BASE_URL=http://host:8000 ANTHROPIC_API_KEY=$KEY claude` works. `/v1/messages/count_tokens`
|
||||
returns the prompt length.
|
||||
- `/v1/responses` maps `instructions`/`input` (strings, messages, `function_call`,
|
||||
`function_call_output`, `reasoning` items) into the chat request; `max_output_tokens`,
|
||||
`reasoning.effort`, `text.format` map to chat fields; undefined fields pass through. The event stream
|
||||
carries `output_index`, `content_index`, `sequence_number`, and opens with `response.created` and
|
||||
`response.in_progress`. A reply cut at `max_output_tokens` is `incomplete`. Function tools a
|
||||
`namespace` groups are named `<namespace>__<name>`; built-in tools and freeform custom tools are
|
||||
left out.
|
||||
- `/completion`, `/completions`, `/infill` are llama-server's own shapes, for llama.vim/llama.vscode.
|
||||
|
||||
## Auth, CORS, errors
|
||||
|
||||
- `--api-key KEY` requires `Authorization: Bearer KEY` or `X-Api-Key: KEY` on every request except
|
||||
`/health`, `/v1/health`, `/ping` and the dashboard page. An empty key is refused at startup.
|
||||
- CORS headers are sent unless `--no-cors`.
|
||||
- Errors are JSON. Over-context is 400 `context_length_exceeded`; a full queue is 429 with
|
||||
`Retry-After`; a capability the model lacks is 501; a guarded field is 400 naming the field and its
|
||||
accepted off value.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Radiance Serve Options]] — the defaults every omitted field takes.
|
||||
- [[Radiance Operations]] — `/health` semantics, metrics, dashboard.
|
||||
- [[frank-archon]] — the harness contract built on this surface.
|
||||
@@ -0,0 +1,142 @@
|
||||
---
|
||||
type: "reference"
|
||||
title: "Radiance Model Families — arch ids, descriptors and recipes"
|
||||
description: "Which radiance plugin claims which model family, and the conversion command and recipe of record for each."
|
||||
tags:
|
||||
- llm
|
||||
- radiance
|
||||
- models
|
||||
- reference
|
||||
timestamp: 2026-10-30
|
||||
---
|
||||
|
||||
# Radiance model families
|
||||
|
||||
The selection key is the exact **(architecture id, quantisation descriptor)** pair, so a family at
|
||||
two weight formats is two `.so` files. A checkpoint's id is its `config.json` `model_type` without
|
||||
underscores (`qwen3_5` → `qwen35`); `rad-convert --arch ID` overrides it. `data/recipes/` is the
|
||||
authority: **read the recipe's header** — it names the exact `rad-convert` line, its `--set` keys
|
||||
and whether it needs calibration.
|
||||
|
||||
## The plugin table
|
||||
|
||||
| plugin | id | quant | what it serves |
|
||||
|---|---|---|---|
|
||||
| `llama_fp8` | `llama` | `fp8_e4m3` | plain LLaMA decoder (MiniCPM5, Llama 3, derivatives reporting `model_type: llama`) from a checkpoint holding E4M3 codes + `weight_scale_inv` |
|
||||
| `llama_dense_fp8` | `llama` | *(empty)* | the same graph from a bf16 checkpoint, linears made block fp8 by a recipe |
|
||||
| `qwen35_bf16` | `qwen35` | *(empty)* | Qwen3.5/3.6/3.8 dense hybrid (GDN + gated attention), bf16 |
|
||||
| `qwen35_fp8` | `qwen35` | `fp8_e4m3` | the same, block-scaled FP8 weights and activations |
|
||||
| `qwen35_w8a8` | `qwen35` | `w8a8_fwht128` | int8 trunk: i8 weights, bf16 scale per 128 of K, 128-wide Hadamard, int8 activations |
|
||||
| `qwen35moe_fp8` | `qwen35moe` | `fp8_e4m3` | `qwen35_fp8` with a routed feed-forward |
|
||||
| `qwen4exp_fp8` | `qwen4exp` | *(empty)* | Qwen4-Exp / Qwen3.8-Flash-Next: a bf16 checkpoint quantised at convert, or Qwen's FP8 release as it ships |
|
||||
| `glm5next` | `glm5next` | *(empty)* | GLM-5.3-Flash: four mHC residual streams, 34 KDA + 11 sparse MLA layers with a DSA indexer, 288 MXFP4 experts top-8, block-FP8 attention/dense at f32 scales, MTP drafter, vision tower |
|
||||
| `hyv3_bf16` | `hyv3` | *(empty)* | Tencent Hy-Memory-30B-A3B bf16; bf16/W4/W8A8/block-FP8/MXFP4 by recipe |
|
||||
| `hyv3_fp8` | `hyv3` | `fp8` | Hy-Memory-30B-A3B-FP8: imported per-tensor E4M3, one dense layer then sigmoid-routed MoE |
|
||||
| `hyv3_gguf` | `hyv3` | `gguf_ftype_15` | a llama.cpp Hy-Memory Q4_K_M GGUF, dequantised and recipe-converted |
|
||||
| `gemma4_bf16` | `gemma4` | `compressed-tensors` | Gemma 4 31B dense (and E2B/E4B at one rank): sliding-window + proportional-RoPE global attention, GELU, softcapped logits; int4/Q8 codes per 128; assistant (shared-KV), DFlash or DFlash2 drafter; vision |
|
||||
| `gemma4` | `gemma4` | *(empty)* | Gemma 4 text from Google's bf16 checkpoint (26B-A4B MoE, dense sizes) with z-lab's DFlash |
|
||||
| `gemma3` | `gemma3` | *(empty)* | Gemma 3 dense (TranslateGemma 27B): sliding-window + global attention, GemmaRMSNorm, GELU; bf16 or int8/MXFP4/asym-int4 linears; DFlash2 drafter |
|
||||
| `lfm2_bf16` | `lfm2` | *(empty)* | LiquidAI LFM2 / LFM2-VL, bf16, one rank |
|
||||
| `embgemma2` | `embeddinggemma2` | *(empty)* | EmbeddingGemma 2: bidirectional encoder with Gemma 4 vision tower and audio conformer, pooled |
|
||||
| `nomicbert` | `nomicbert` | *(empty)* | nomic-embed-text-v2-moe: bidirectional post-norm encoder, 8 experts top-2 on every second layer, mean-pooled |
|
||||
| `nemotron35_embed` | `mistral3` | *(empty)* | Nemotron 3.5 Embed 1B/8B and Rerank 8B: bf16 text/image retrieval with a Pixtral tower |
|
||||
| `qwenimage21` | `qwenimage21` | *(empty)* | Qwen-Image 2.1 / 2.1-Turbo image generator and editor (diffusers pipeline) |
|
||||
| `krea2` | `krea2` | *(empty)* | Krea 2 Raw/Turbo: Qwen3-VL-4B tapped layers, 12.9B single-stream DiT, Wan 2.1 VAE |
|
||||
| `sdxl` | `sdxl` | *(empty)* | Stable Diffusion XL and finetunes (Illustrious, NoobAI, Pony), single-file checkpoint |
|
||||
| `flux3` | `flux3` | *(empty)* | FLUX.3 [klein] and base: image generation and edits from up to ten pictures |
|
||||
| `flux3distilled` | `flux3distilled` | *(empty)* | the same plugin under the step-distilled release's own pipeline name |
|
||||
| `anima` | `anima` | *(empty)* | Anima Base/Aesthetic/Turbo from the release's three single files |
|
||||
| `qwen3asr` | `qwen3asr` | *(empty)* | Qwen3-ASR: Whisper-style audio encoder over a dense Qwen3 decoder, speech to text |
|
||||
| `minimaxmusic3` | `minimaxmusic3` | *(empty)* | MiniMax-Music3: Qwen3-8B frame model, depth decoder, flow-matching DiT, DAC vocoder |
|
||||
| `breeze` | `breeze` | *(empty)* | Breeze-TTS-2: T5Gemma 2 text encoder, Qwen3 backbone, 16-codebook depth decoder, Qwen3-TTS 12 Hz codec |
|
||||
| `fishomni` | `fishomni` | `bsgemm-moe` | Fish Audio drama-3: Qwen3.5-35B-A3B backbone, block-FP8 experts at f32 scales, 13-layer codebook tower, Fish S2 DAC/codec |
|
||||
| `minimaxh3` | `minimaxh3` | *(empty)* | MiniMax-H3 video with stereo sound: Qwen3-VL-32B text encoder, 33B DiT, tiled ViT decoder, BigVGAN audio |
|
||||
| `ltx2` | `ltx2` | *(empty)* | LTX-2.5 base/distilled: Gemma 4 LM + text connectors, duration head, audio-video DiT, causal video VAE, audio VAE |
|
||||
|
||||
## Recipes of record, by family
|
||||
|
||||
`$SRC` is a downloaded checkpoint; `$CALIB` is a calibration directory (see
|
||||
[[Radiance Conversion]]). Paths are shortened to the recipe name.
|
||||
|
||||
| target | command |
|
||||
|---|---|
|
||||
| Qwen3.8-27B bf16 | `rad-convert $SRC -o q38-27b-bf16.rad --recipe data/recipes/q38-27b-bf16.recipe` |
|
||||
| Qwen3.8-27B block fp8 + DFlash2 | `rad-convert $SRC -o q38-27b-fp8.rad --recipe data/recipes/q38-27b-fp8block-df2.recipe --draft-model $DFLASH2 --set quantization_config.quant_method=fp8 --set quantization_config.fmt=e4m3 --set "quantization_config.weight_block_size=128 128" --set quantization_config.activation_scheme=dynamic` |
|
||||
| Qwen3.8-27B MXFP4 + DFlash2 | `rad-convert $SRC -o q38-27b-mxfp4.rad --recipe data/recipes/q38-27b-mxfp4-df2.recipe --draft-model $DFLASH2 --set quantization_config.quant_method=fp8 --set quantization_config.fmt=e4m3` |
|
||||
| Qwen3.8-27B w4a16 (an int4 release) | `rad-convert $SRC -o q38-27b-w4a16.rad --recipe data/recipes/q38-27b-w4a16.recipe` |
|
||||
| Qwen3.8-27B Quark AWQ-MXFP4 | `rad-convert $SRC -o q38-27b-quark.rad --recipe data/recipes/q38-27b-quark-mxfp4-df2.recipe --draft-model $DFLASH2` |
|
||||
| Qwen3.8-27B ParoQuant int4/int5/MXFP4 | `rad-convert $SRC -o q38-27b-paro.rad --recipe data/recipes/q38-27b-paro-mxfp4-df2.recipe --draft-model $DFLASH2` |
|
||||
| Qwen3.8-27B NVFP4 (compressed-tensors or ModelOpt) | `--recipe data/recipes/q38-27b-nvfp4-df2.recipe` or `q38-27b-modelopt-nvfp4.recipe`; every NVFP4 weight must take a rule (no kernel serves NVFP4) |
|
||||
| Qwen3.8-Flash-Next bf16 reference | `rad-convert $SRC -o flashnext-bf16.rad --max-ctx 4096` (no recipe; served with `--weights-disk-tier`) |
|
||||
| Qwen3.8-Flash-Next four-bit production | `CALIB=$CALIB rad-convert $SRC -o w4nl64-i8lm.rad --recipe data/recipes/qwen4exp-w4nl64-i8-hc8m.recipe` (GPTQ experts + int8 trunk) |
|
||||
| Qwen3.8-Flash-Next uncalibrated four-bit | `--recipe data/recipes/qwen4exp-w4.recipe` (rtn, no calibration) |
|
||||
| Flash-Next from an AutoRound int4 release | `--recipe data/recipes/qwen4exp-autoround-i8-hc8m.recipe` |
|
||||
| Gemma 4 31B w4a8 (no calibration) | `--recipe data/recipes/gemma4-31b-dense-bf16-w4a8.recipe --set quantization_config.quant_method=compressed-tensors --draft-model $ASSISTANT` |
|
||||
| Gemma 4 31B w4a8 + DFlash / DFlash2 | `gemma4-31b-dense-bf16-w4a8-df.recipe` / `-w4a8-df2.recipe`, same `--set` |
|
||||
| Gemma 4 31B Q8 grid | `gemma4-31b-dense-bf16-q8-df.recipe` |
|
||||
| Gemma 4 26B-A4B + DFlash | `gemma4-26b-a4b-df.recipe`; `-pad768` when the experts were padded to 768 (down may then be block fp8) |
|
||||
| Gemma 4 E4B (Nemotron Bolt) | `gemma4-e4b-bf16-df.recipe`, `-i8g-`, `-a4-`, `-w4a8-`, `-w8a8-` variants |
|
||||
| TranslateGemma 27B | `translategemma-27b-i8a8.recipe` (calibration container), then `CALIB=$CALIB translategemma-27b-mxfp4-gptq.recipe` (or `-df2`) |
|
||||
| GLM-5.3-Flash | `rad-convert $AMD_QUARK -o glm53.rad --recipe data/recipes/glm53-mxfp4.recipe` |
|
||||
| Hy-Memory-30B-A3B | `hy-memory-30b-a3b-bf16-fp8.recipe`, `-mxfp4`, `-w4`, `-w8`, `hy-memory-30b-a3b-fp8.recipe`, `hy-memory-30b-a3b-gguf-q4k-to-w4.recipe` |
|
||||
| MiniCPM5-2B | `minicpm5-2b-dspark.recipe` (or `-bf16-dspark`) with `--draft-model $DSPARK` |
|
||||
| Qwen3.6-35B-A3B | `q36-35b-a3b-fp8.recipe` |
|
||||
| Qwen-Image 2.1 | `qwen-image-2.1-fp8.recipe` (Turbo uses the same file) or `-bf16` |
|
||||
| Krea 2 | `krea-2-fp8.recipe`, `krea-2-bf16.recipe`, `krea-2-fp8-unscaled.recipe` for a plain-FP8 finetune file |
|
||||
| FLUX.3 [klein] | `flux-3-klein-fp8.recipe` |
|
||||
| SDXL / finetunes | `sdxl-bf16.recipe --tokenizer <openai/clip-vit-large-patch14>/tokenizer.json` |
|
||||
| Anima | `anima-fp8.recipe` (or `-bf16`) + `--add` ×2 + `--config` + `--tokenizer2` + `--arch anima --set anima.variant=aesthetic` |
|
||||
| LTX-2.5 | `ltx-2.5-fp8.recipe` (or `-bf16`); full DiT: symlink a folder whose `transformer/` is `transformer_full/` and `--set scheduler.use_dynamic_shifting=1 --set scheduler.shift_terminal=0.1` |
|
||||
| MiniMax-H3 / FastH3 | `minimax-h3-fp8.recipe`, `minimax-h3-w8a8.recipe`, `minimax-h3-curve-*` for curve-form checkpoints |
|
||||
| MiniMax-Music3 | `minimax-music3-bf16.recipe` |
|
||||
| Breeze-TTS-2 | `breeze-tts2-bf16.recipe` |
|
||||
| drama-3 | `drama3-fp8.recipe` |
|
||||
| nomic-embed-v2-moe | `nomic-embed-v2-moe-bf16.recipe` |
|
||||
| Unlimited-OCR | `unlimited-ocr-bf16.recipe` |
|
||||
| Qwen3-ASR | no container: serve the checkpoint directory as it ships |
|
||||
| EmbeddingGemma 2, Nemotron Embed/Rerank | serve the checkpoint, or convert bf16 with no recipe |
|
||||
|
||||
A checkpoint served at its own precision needs no container at all:
|
||||
`radiance --model /srv/models/hf/Qwen/Qwen3.8-27B-FP8`. Convert when you want a tier only a container
|
||||
has (quantised encodings, a drafter, a second vocab) or a smaller file.
|
||||
|
||||
## Family-specific traps
|
||||
|
||||
- **`head_dim` is read, never derived.** On the Qwen hybrids it is 256 while `hidden_size /
|
||||
num_attention_heads` is 213; a derived value silently breaks projection widths, KV page size and the
|
||||
attention kernel's band.
|
||||
- **A fused projection shards by part, not by contiguous rows.** A tensor-parallel rank owns a slice
|
||||
of *each* stacked projection in a fused `[gate|up]` or `[q|k|v|z]`; a contiguous slice would hand
|
||||
rank 0 the whole gate, and the model would produce fluent wrong text. This is declared through
|
||||
`RadWeightDecl::row_parts`, never inferred from the container.
|
||||
- **Gemma 4 26B-A4B:** the routed experts' `gate_up` may be block fp8, but `down` stays bf16 unless the
|
||||
experts were padded to 768 — hence the `-pad768` recipe.
|
||||
- **GLM-5.3-Flash:** every weight is served as stored except the latent attention's small `kv_b`,
|
||||
decoded to bf16. Its KV cache is bf16 only (the plugin refuses `fp8` at startup), and at 172 GiB it
|
||||
needs `--tp 8`.
|
||||
- **TranslateGemma 27B:** the checkpoint's template takes structured turns naming two languages and
|
||||
refuses the plain turn the server renders at startup. The container therefore serves a plain Gemma 3
|
||||
template, and a client sends on `/v1/completions` the prompt the checkpoint's own template renders.
|
||||
- **ParoQuant containers made before 1.5.0 are refused at startup.** Convert again with the old
|
||||
container as `--reuse`; every tensor that did not change is copied.
|
||||
- **SDXL carries no tokeniser.** Pass CLIP's `tokenizer.json` beside the single file. A Krea 2
|
||||
finetune shipped as one DiT file in Krea's key names is rewritten into a directory whose
|
||||
`transformer/` holds it (`scripts/krea2_native.py`), with the release's other parts copied in.
|
||||
- **Anima** ships as three single files: `--add` the text encoder and VAE, `--config` a Qwen3-0.6B-Base
|
||||
directory (config + tokenizer), `--tokenizer2` ComfyUI's T5 tokenizer, `--arch anima`,
|
||||
`--set anima.variant=…`.
|
||||
- **drama-3's codec is `codec.pth`**, a `torch.save` file read as data, never unpickled.
|
||||
- **Generators that load LoRA files declare their module aliases**; the checkpoint and a ComfyUI port
|
||||
disagree in ways that load without error and generate garbage. The per-plugin alias table is in
|
||||
`docs/ARCHITECTURES.md` ("Adapter key names of the generators").
|
||||
- **Licences travel with the checkpoint, not the engine.** Qwen-Image 2.1 is non-commercial;
|
||||
MiniMax-H3's licence excludes the EU, UK, Republic of Korea and USA; drama-3's community licence
|
||||
needs a commercial licence to serve speech to others, and cloning a real person needs consent;
|
||||
Breeze-TTS-2 is research/non-commercial.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Radiance Conversion]] — the tool, recipe language and quantisers used above.
|
||||
- [[Radiance Serve Options]] — the flags each family's compose file uses.
|
||||
- [[Radiance Architecture Plugins]] — writing a plugin for a family not in this table.
|
||||
- [[Quantization Production]] — the fleet's campaign and gates.
|
||||
@@ -0,0 +1,240 @@
|
||||
---
|
||||
type: "howto"
|
||||
title: "Radiance Operations — running, reading, restarting"
|
||||
description: "Running a radiance server: the container contract, host prerequisites, readiness, ledgers and safe restarts."
|
||||
tags:
|
||||
- llm
|
||||
- radiance
|
||||
- operations
|
||||
- howto
|
||||
timestamp: 2026-10-30
|
||||
---
|
||||
|
||||
# Radiance operations
|
||||
|
||||
This is the engine-level runbook. The fleet's deployed shape, its unit and its deploy path live on
|
||||
[[frank-archon]]; the flags themselves on [[Radiance Serve Options]].
|
||||
|
||||
## Install
|
||||
|
||||
```sh
|
||||
git clone https://codeberg.org/StillDeadcode/radiance.git && cd radiance
|
||||
./build.sh # configure, build for the cards present, run fast tests
|
||||
./build.sh --no-hip # host backend only
|
||||
cmake -S . -B build -G Ninja -DRAD_GPU_TARGETS='gfx1201' # cards this machine lacks
|
||||
cmake --install build # after -DCMAKE_INSTALL_PREFIX=/opt/radiance at first configure
|
||||
```
|
||||
|
||||
Needs Linux, git, a C++20 compiler, CMake 3.21+, Ninja; for device kernels ROCm 10.1 or 7.2
|
||||
(`hipcc`, `ROCM_PATH` if not `/opt/rocm`).
|
||||
|
||||
- **Name the prefix at the first configure.** The default plugin path `<prefix>/share/radiance` is
|
||||
compiled in from that configure; `cmake --install build --prefix DIR` alone gives a tree that looks
|
||||
for its plugins elsewhere.
|
||||
- The prefix gets `bin/`, `include/radiance/`, `lib/cmake/radiance/`, `libexec/radiance/`,
|
||||
`share/radiance/` (plugins and recipes) and `share/doc/radiance/`.
|
||||
- With ROCm 10.1 the build turns VOPD pairing off because its LLVM 24 pairs in heap-address order, so
|
||||
two builds of one source can differ. The Docker image's compiler has the fix and keeps pairing
|
||||
(faster fp8 prefill GEMMs); declare such a compiler with `-DRAD_HIP_VOPD_REPEATABLE=ON`. ROCm 7.2 is
|
||||
unaffected.
|
||||
|
||||
Run without Docker:
|
||||
|
||||
```sh
|
||||
export RADIANCE_HOME=$PWD/build/radiance_home # plugins; an install finds its own
|
||||
build/bin/radiance --model /srv/models/rad/minicpm5-2b-fp8.rad --port 8000 <flags>
|
||||
build/bin/radiance --model StillDeadcode/minicpm5-2b-fp8 --port 8000 <flags>
|
||||
```
|
||||
|
||||
`--model` also takes a repo id: the server runs `scripts/hfget.sh` into `--download-dir`, serves the
|
||||
one `.rad` at the repo's top level, or the directory as a checkpoint when it holds none. A repo with
|
||||
several containers is refused, naming them. Every start checks the hub; a start that cannot reach it
|
||||
serves the copy a complete earlier fetch left, and a transfer that fails part-way stops startup.
|
||||
|
||||
## Container
|
||||
|
||||
The published image is `stilldeadcode/radiance`, kernels built for `gfx1201`. The host needs Docker
|
||||
and an `amdgpu` driver supporting ROCm 10.1 — **not** ROCm itself.
|
||||
|
||||
```sh
|
||||
docker run -d --name radiance --network host --init --ulimit memlock=-1 \
|
||||
--device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined \
|
||||
--group-add "$(getent group render | cut -d: -f3)" \
|
||||
--group-add "$(getent group video | cut -d: -f3)" \
|
||||
-v /srv/models/rad:/models:ro \
|
||||
radiance --model /models/m.rad --tp 2 --host :: --port 8000
|
||||
```
|
||||
|
||||
Each option is load-bearing:
|
||||
|
||||
| option | why |
|
||||
|---|---|
|
||||
| `--device /dev/kfd --device /dev/dri`, `render`/`video` groups | the cards |
|
||||
| `--security-opt seccomp=unconfined` | Docker's default profile refuses io_uring (a disk-tier n-gram table's reads) and the NUMA calls the ROCm runtime makes for host memory |
|
||||
| `--init` | an init process as PID 1 forwards SIGTERM; the engine finishes its step and releases the cards |
|
||||
| `--ulimit memlock=-1` | a host-side pinned n-gram copy can be tens of GiB, far past Docker's default |
|
||||
| `--network host` | the server binds the host's addresses directly, IPv4 and IPv6, no NAT |
|
||||
|
||||
The image is `FROM scratch` around the stripped install tree: busybox as the only shell, plus `curl`,
|
||||
`jq` and a CA bundle for `hfget.sh`; **no Python, no package manager**. Tools (`rad-convert`,
|
||||
`rad-info`, `rad-kbench`, `rad-tune`, `rad-schemas`, `rad-tokcheck`) are on `PATH`; recipes are in
|
||||
`/opt/radiance/share/radiance/recipes/`. Its entrypoint is `radiance`. It sets
|
||||
`GPU_MAX_ALLOC_PERCENT=100` and `GPU_MAX_HEAP_SIZE=100` because the HIP runtime's defaults cap one
|
||||
allocation and the heap below what the placement plan hands out.
|
||||
|
||||
Build your own with `docker/build.sh` (`-t 'gfx1200;gfx1201'` for targets, `-j` compile jobs, `-E`
|
||||
to skip tests, `-s host` to build remotely). The context is `git archive` of the commit, so the image
|
||||
holds only committed content. The `rootfs` stage fails the build if any ELF cannot be resolved inside
|
||||
a chroot of the tree.
|
||||
|
||||
Compose: one file per model plus `common.yaml`, copied to the serving host with an `.env` setting the
|
||||
image, models directory (mounted read-only at `/models`), API key, port, user and GPU groups. Every
|
||||
model listens on the same port, and most use both cards, so **run one at a time**. The models
|
||||
directory must be on a local filesystem that does direct I/O (ext4, XFS, btrfs); on btrfs with
|
||||
compression, a container copied into a `chattr +C` directory is stored uncompressed and direct reads
|
||||
work.
|
||||
|
||||
A plugin of your own is mounted and put first on the search path; nothing in the image changes:
|
||||
|
||||
```sh
|
||||
-v /srv/radiance-plugins:/plugins:ro -e RADIANCE_HOME=/plugins:/opt/radiance/share/radiance
|
||||
```
|
||||
|
||||
## Host prerequisites (upstream's two-R9700 checklist)
|
||||
|
||||
| item | setting |
|
||||
|---|---|
|
||||
| firmware | Above 4G Decoding on (Resizable BAR needs it); CSM off (pure UEFI) |
|
||||
| kernel command line | `iommu=pt amdgpu.ppfeaturemask=0xfff7ffff amdgpu.ras_enable=0` |
|
||||
| power/clock | OverDrive bit needed for `ppfeaturemask`; apply a power cap only while no process holds `/dev/kfd` |
|
||||
| memlock | `--ulimit memlock=-1` (container) / `LimitMEMLOCK=infinity` (systemd) when host memory is pinned |
|
||||
| two engines, one tmpfs table | `amdgpu.no_system_mem_limit=1`, or the second pin can be refused |
|
||||
| storage | a local fs with O_DIRECT for the container; a tmpfs sized for the table for `--ngram-placement ram=DIR` |
|
||||
|
||||
Why each: `iommu=pt` keeps DMA identity so the cards can reach each other's memory and pinned host
|
||||
pages; `ppfeaturemask` enables the power controls; `ras_enable=0` turns VRAM ECC off (ECC reserves
|
||||
about 2 GiB a card). Verify peer access with a rate in `lspci`/the card tables — a missing cell means
|
||||
no peer access; check `iommu=pt` and the BAR size. The fleet's own boot-line law for Dankclaw
|
||||
(`iommu=pt`, tracked in the repo) is on [[dankclaw-iommu-translation-off]]; Dankserver's cards are a
|
||||
different box and its posture is on [[Dankserver System Specs]].
|
||||
|
||||
## Readiness: the receipt, then `/health`
|
||||
|
||||
Startup states every budget **before anything is allocated**, then loads. `/health` (and
|
||||
`/v1/health`, `/ping`) answers only once the load completes, so a bare health gate is honest. While it
|
||||
loads, expect 503 for minutes; do not restart over it. `--live` draws the live view (redirected, one
|
||||
`greppable key=value` line). `/diagnostics` returns the startup report as JSON — **same-machine-only
|
||||
without an API key** — with every option and its source and the hardware read again now.
|
||||
|
||||
Read the receipt in this order:
|
||||
|
||||
1. `elastic` — what the experts and KV cache share after non-expert weights.
|
||||
2. The split: `experts … / kv cache …`, and `unclaimed` if any.
|
||||
3. `kv: the pool backs N % of the worst case … M full-length sessions` — the referee for whether the
|
||||
demand signal (`--max-model-len` × `--max-num-seqs`) is funded.
|
||||
4. `activation arena` and what each size lends; `prefill staging`; `hardware queues` and their VRAM.
|
||||
5. `host_pool` opened/used, `vram_weights`, the PLE or n-gram placement, the vision tower's placement.
|
||||
6. `device: the N cards reach each other's memory directly` or the failed pairs;
|
||||
`expert prefetch on/off`; `chunk geometry`; `speculation:`; `chained launches`; `draft trees`.
|
||||
7. `card N settled at … free` — memory taken after the budget read comes out of headroom; a warning
|
||||
names the `--gpu-headroom-mib` that clears it.
|
||||
|
||||
A boot refusal prints the reason and exits nonzero; when it happens inside `podman run`, the
|
||||
container stops at once and the unit's restart policy re-runs the same refusal. A unit in
|
||||
`activating (auto-restart)` with a fast nonzero exit is therefore a **preflight/refusal or a boot
|
||||
refusal**, not an engine fault: read the journal line and the argv, not the GPU.
|
||||
|
||||
## Lifecycle and restart
|
||||
|
||||
- SIGTERM finishes the current step and releases the cards. In a container, `--init` forwards it;
|
||||
`podman stop -t N`/`docker stop` is the graceful path.
|
||||
- A systemd unit should `exec podman wait` (or run in the foreground) so the script's lifetime is the
|
||||
container's: a crash is a nonzero exit and the restart policy is real. Set `TimeoutStopSec` above
|
||||
the stop grace, or systemd records `Failed with result timeout` on a healthy stop.
|
||||
- `Restart=always` rather than `on-failure` for a server whose front end can exit cleanly after a
|
||||
worker fault.
|
||||
- Model load is slow (minutes cold). The deployer's LLM wait budget is on [[Deployment Flow]]; a
|
||||
container restart loop during a legitimate load is worse than waiting.
|
||||
|
||||
## Two engines on one set of cards
|
||||
|
||||
- `ROCR_VISIBLE_DEVICES` picks each engine's cards explicitly.
|
||||
- `--weight-prefetch off`: a weight read beside a collective puts a queue switch into every decode
|
||||
step when another server shares the cards.
|
||||
- `--ngram-placement ram=DIR`: two engines on separate card sets hold one table in host memory
|
||||
(`DIR/ngram-<id>.tab` in a tmpfs, `MAP_SHARED`, pinned). The `<id>` hashes the container path, size,
|
||||
mtime and the table's place, so a file never serves another model. The first engine fills it; a
|
||||
second verifies samples and only pins it; the last to leave deletes it. The driver counts a shared
|
||||
page against every process that pins it, so the pinned total is twice the table.
|
||||
- The engine prefers discrete cards (all of the first one's architecture), integrated only when
|
||||
there is nothing else; on integrated graphics device memory is host memory and the budget leaves an
|
||||
eighth of host RAM to the host.
|
||||
|
||||
## Logs and diagnostics
|
||||
|
||||
- `-v` / `-vv`; `RADIANCE_LOG` sets the level without a flag. `RADIANCE_LOG_STEPS=1` prints every
|
||||
step's token count (how a chunk geometry is confirmed in practice).
|
||||
- The engine writes to stderr. Under a container, the journal holds only the container's own output;
|
||||
a rootless `podman logs` covers only the current instance across recreations.
|
||||
- `/metrics` is Prometheus text: engine steps by kind and their seconds, decode/prefill rates,
|
||||
prefill chunks held back, steps held by interleave, draft acceptance and depth, prompt-lookup
|
||||
proposals held, KV bytes and the tokens they address, stream misses and bytes, prefix hits and
|
||||
evictions, preemptions and queue depth. `/stats`,
|
||||
`/graph`, `/sessions` back the dashboard. `scripts/statsdump.py` records `/stats` to JSONL with
|
||||
time and gap, for a machine with no scraper (its key comes from `--api-key` or
|
||||
`RADIANCE_API_KEY`).
|
||||
- `RADIANCE_PROFILE_EVERY`, `RADIANCE_DUMP_BUF`/`RADIANCE_DUMP_DIR`, `RADIANCE_OP_ORACLE`(+) and the
|
||||
`RADIANCE_DEBUG_*` family are debug instruments; they are not deployment settings.
|
||||
|
||||
## Verifying a deploy (fleet law)
|
||||
|
||||
`deploy.sh` green is not proof of what is running. A leftover systemd drop-in or a stale container
|
||||
identity can serve a different model at a different shape while `/health` is 200. After deploying
|
||||
any recently-benched service:
|
||||
|
||||
1. `systemctl show <svc> -p DropInPaths` — must be empty.
|
||||
2. `tr '\0' ' ' </proc/$(pidof radiance)/cmdline` (or the container's `Config.Cmd`) — the argv you
|
||||
meant.
|
||||
3. The boot ledger — the shape you meant.
|
||||
4. A behavioural probe — `/v1/models` for the served id, one chat turn, one tool call, and a vision
|
||||
probe with a solid colour and an exact expected word.
|
||||
|
||||
`/v1/models` is an echo on some runtimes; on radiance it is authoritative for the alias, but the
|
||||
argv and the ledger are what prove the payload and shape. The three-pin revert discipline for a
|
||||
payload swap is on [[Model Swap and Rollback]]; the live deployment is [[frank-archon]].
|
||||
|
||||
## Security posture
|
||||
|
||||
- `--api-key` requires a bearer token or `X-Api-Key` on every route except `/health`, `/v1/health`,
|
||||
`/ping` and the dashboard; the key is a command-line argument, so it is readable in
|
||||
`/proc/<pid>/cmdline` by any local user of that host. On a single-tenant inference box the network
|
||||
boundary is the real control; do not run one where untrusted local users share the host.
|
||||
- `/diagnostics` is same-machine-only without a key, because it prints the whole configuration.
|
||||
- `--security-opt seccomp=unconfined` is a genuine widening, required by io_uring and the ROCm
|
||||
runtime's NUMA calls; it belongs on a dedicated model host, not a general-purpose one.
|
||||
- Mount model directories read-only; a container with `/models` writable adds nothing a serve needs.
|
||||
- A control vector can remove refusal behaviour. Loading one is a safety decision, not a
|
||||
performance one; the flag is `--control-vector`, and `--control-vector disabled` is the default.
|
||||
|
||||
## Failure signatures at startup
|
||||
|
||||
| what you see | what it is |
|
||||
|---|---|
|
||||
| unknown/retired flag, or a flag refused by name | a boot refusal — the pin does not have it; change the trio, not the invocation |
|
||||
| "no plugin claims `<arch>` at `<quant>`; served at: …" | the (id, descriptor) pair has no plugin; fix the descriptor with `--set`/convert |
|
||||
| "reports ABI N, engine M" | rebuild the plugin against this engine |
|
||||
| budget refusal / `host_pool` short | the shape does not fit this card/host; lower demand or pin more host RAM |
|
||||
| `--ngram-placement ram` refuses | memlock short; raise it or use `auto`/`disk` |
|
||||
| `--kv-cache-dtype fp8_v4`/`k6_v4` refused, plugin named | no kernels for this geometry |
|
||||
| `--p2p on` refuses, pairs named | the driver has no peer access; use `auto`/`off` |
|
||||
| first launch of a never-run kernel faults | too little `--gpu-headroom-mib` for its code object |
|
||||
| unit loop, no container, no engine log | a preflight refusal (TTM/memlock/digest/model) re-run by `Restart=always` |
|
||||
|
||||
## Related
|
||||
|
||||
- [[Radiance Serve Options]] — every knob and its law.
|
||||
- [[Radiance HTTP API]] — readiness and the wire.
|
||||
- [[Radiance Tools]] — how to corroborate a receipt.
|
||||
- [[frank-archon]] — the fleet's radiance unit, guard chain and Eras.
|
||||
- [[Model Swap and Rollback]] — the swap discipline this page's verification serves.
|
||||
- [[Deployment Flow]] — how a committed change reaches the host.
|
||||
@@ -0,0 +1,371 @@
|
||||
---
|
||||
type: "howto"
|
||||
title: "Radiance Plugins — kernel libraries and quantisers"
|
||||
description: "Writing radiance kernel and quantiser plugins: the ABI, kernel rows, hooks, encodings, and the build path."
|
||||
tags:
|
||||
- llm
|
||||
- radiance
|
||||
- plugins
|
||||
- howto
|
||||
timestamp: 2026-10-30
|
||||
---
|
||||
|
||||
# Radiance plugins — kernels and quantisers
|
||||
|
||||
Three kinds of `.so` extend the engine, each exporting a few C symbols and linking **nothing of
|
||||
radiance's** (they compile against the installed `abi/` headers):
|
||||
|
||||
| kind | value | contract |
|
||||
|---|---|---|
|
||||
| kernel library | `RAD_PLUGIN_KERNEL` (1) | ops for some hardware: a card family, the CPU, an oracle |
|
||||
| architecture | `RAD_PLUGIN_ARCH` (2) | one model family at one weight format — see [[Radiance Architecture Plugins]] |
|
||||
| quantiser | `RAD_PLUGIN_QUANT` (3) | weights → encoding planes, for `rad-convert` |
|
||||
|
||||
In-tree plugins use the same interface a third party does: each builds with `rad_add_plugin` and is
|
||||
found and ranked by what it declares, never by its name. `ctest -L oot` rebuilds every bundled
|
||||
plugin out of tree against an install.
|
||||
|
||||
## Identity and ABI
|
||||
|
||||
```c
|
||||
#include "rad_abi.h" /* required; pulls rad_types.h and rad_encoding.h */
|
||||
|
||||
uint32_t rad_plugin_abi_version(void) { return RAD_ABI_VERSION; }
|
||||
|
||||
const RadPluginInfo* rad_plugin_info(void) {
|
||||
static const RadPluginInfo i = {
|
||||
RAD_PLUGIN_KERNEL, "mykernels", "0.1", "a GEMM or two", "gfx1100",
|
||||
};
|
||||
return &i;
|
||||
}
|
||||
```
|
||||
|
||||
- `RAD_ABI_VERSION` (31 at this commit) is bumped on any shape change in a header a plugin compiles
|
||||
against. Return the macro; a mismatch is refused at load with both numbers printed.
|
||||
- `build_target` is printed, not trusted: the targets the library was built for (`"gfx1100"`,
|
||||
`"gfx11-generic"`, several separated by spaces/commas), `"host"`, or `""` for portable. The engine
|
||||
checks the library's actual code objects instead (§"A library for a new card").
|
||||
- Optional exports, resolved by exact name: `rad_plugin_open` (return `RAD_E_UNSUPPORTED` to decline
|
||||
this machine and unload), `rad_plugin_close`, `rad_kernel_concurrent`, `rad_kernel_collective`,
|
||||
`rad_plugin_reference`, `rad_kernel_reference`, `rad_kernel_chain`.
|
||||
|
||||
## What a kernel library exports
|
||||
|
||||
```c
|
||||
int rad_kernel_schema_count(void);
|
||||
const RadOpSchema* rad_kernel_schema_at(int i);
|
||||
int rad_kernel_count(void);
|
||||
const RadKernelInfo* rad_kernel_at(int i);
|
||||
```
|
||||
|
||||
A row declares one op implementation and what it can serve:
|
||||
|
||||
```c
|
||||
static const RadConstraint c_gemm[] = {
|
||||
RAD_CLE("M", 64), /* M <= 64 */
|
||||
RAD_CDIV("K", 128), /* K divisible by 128 */
|
||||
RAD_CIN("dtype", "fp8a8 w4a8") /* dtype in that set */
|
||||
};
|
||||
|
||||
static const RadKernelInfo k[] = {{
|
||||
.name = "mygemm_m64", .op = "gemm_nt_q", .family = "gemm",
|
||||
.computes = "row-major A times column-major B, fp8 in, bf16 out",
|
||||
.shape = "M<=64, K%128==0",
|
||||
.dtypes = "fp8a8, w4a8",
|
||||
.domain = RAD_DOMAIN_DEVICE,
|
||||
.priority = 50,
|
||||
.constraints = c_gemm, .n_constraints = 3,
|
||||
.launch = my_launch,
|
||||
}};
|
||||
```
|
||||
|
||||
- Operators: `RAD_CEQ`, `RAD_CLE`, `RAD_CGE`, `RAD_CDIV`, `RAD_CIN`. The engine takes bucket
|
||||
boundaries from `LE`/`GE`/`EQ` and resolves once per band; `DIV` and `IN` are not intervals.
|
||||
- **Declare only the band you serve.** The engine believes the claim; a kernel that says `M <= 2048`
|
||||
but is correct to 64 is wrong at concurrency.
|
||||
- `device.*` keys are matched against what the engine measured about the ranks' cards, not against
|
||||
op parameters: `RAD_CEQ(RAD_FACT_P2P, 1)` means the kernel touches another card's memory. libr4d
|
||||
pairs every collective with a `_host` twin requiring `device.p2p == 0`, so exactly one matches any
|
||||
machine.
|
||||
- **Priority breaks ties within one plugin**, highest first, then declaration order. **Between
|
||||
plugins the hierarchy is absolute**: the first library in `--kernels` order with any matching row
|
||||
supplies the op, however specialised a later library's row is. That makes a user's library an
|
||||
override, not a competitor.
|
||||
- A derivative parameter (`RAD_DERIVED`) is supplied by the kernel; a constraint on a key the
|
||||
geometry omits does not hold during matching, and the winner's `RAD_C_EQ` writes it into the
|
||||
resolved geometry. `RAD_PROLE_SEQ_CHUNK` marks the parameter the scheduler must cut steps on;
|
||||
`RAD_PROLE_CAPACITY` marks a count the kernel only bounds itself by.
|
||||
|
||||
## Operands and extents
|
||||
|
||||
`RadArgs.t` is positional in the schema's order; each `RadTensor` carries the extent the caller
|
||||
narrowed it to, usually smaller than the buffer. Size loops and grids from those extents, never from
|
||||
a declared maximum. Where an op takes `cu_seqlens`, derive the sequence count from its own length
|
||||
(`n = len - 1`) and each sequence's rows from `cu[n]` and `cu[n+1]`.
|
||||
|
||||
An absent optional operand is a **null `RadTensor.data`**, not a shorter operand count.
|
||||
`rad_arg_in(a, i)` fetches one correctly; `rad_tensor_rows`, `rad_tensor_lastdim`,
|
||||
`rad_tensor_row_pitch`, `rad_args_geti_or`, `rad_args_getf_or`, `rad_args_gets_or` are the other
|
||||
everyday helpers in `rad_plugin.h`.
|
||||
|
||||
## Optional hooks
|
||||
|
||||
| hook | meaning |
|
||||
|---|---|
|
||||
| `init` / `fini` | per-resolved-instance state; `RadArgs.instance` hands it back. `init` runs once per (op, band, domain) after selection and before load |
|
||||
| `scratch` | bytes of scratch one launch needs; sized at declare from the band's upper bound |
|
||||
| `layout` / `relayout` | the kernel stores this weight in its own order, and how the loader produces it from the canonical planes |
|
||||
| `unrelayout` | the inverse, so a tool holding stored bytes can compare against a dense oracle |
|
||||
| `opd_shape` | size and fill every operand, so a tool can call the kernel cold |
|
||||
| `replaces` | the chain of conventional ops this kernel computes in one launch |
|
||||
| `tunables` / `tune_valid` | the instantiation axes, and which combinations are legal at a shape |
|
||||
|
||||
`replaces` must be **exact** and is compared byte for byte against the chain — a kernel that is only
|
||||
close publishes nothing. `tunables` declares axes and legal values, not named variants; the core
|
||||
takes the cross product, and `deflt` must be launchable (the loader refuses the plugin otherwise).
|
||||
`rad-tune` measures survivors and writes the cache the selector reads; the chosen point arrives as
|
||||
ordinary parameters (`rad_args_geti("sk")`), with no variant index.
|
||||
|
||||
`layout` answers: `RAD_OK` (own form, `relayout` will produce it), `RAD_E_UNSUPPORTED` (stored as
|
||||
is — the common case, and the only form that can be read in place from the file/mapped tier),
|
||||
`RAD_E_DTYPE` (this encoding is not mine — a declare-time error), `RAD_E_SHAPE`, or `RAD_LAYOUT_ANY`
|
||||
(reads the bytes without interpreting them; claims no storage). A weight only `LAYOUT_ANY` readers
|
||||
touch is stored as it is.
|
||||
|
||||
## What a launch may do
|
||||
|
||||
On a card the `RadStream` is **the engine's own AQL queue**, not a `hipStream_t`. The engine binary
|
||||
defines and serves these HIP entry points:
|
||||
|
||||
- `hipLaunchKernel` (and `<<<>>>`, `hipLaunchKernelGGL`);
|
||||
- `hipMemcpyAsync`, `hipMemcpy2DAsync`, `hipMemsetAsync`;
|
||||
- `hipStreamSynchronize` (not from a launch — a launch must not synchronise);
|
||||
- `hipMemset` (runs on the engine's queue and waits);
|
||||
- `hipGetLastError` / `hipPeekAtLastError`, with HIP's semantics.
|
||||
|
||||
**Nothing else in HIP takes an engine stream.** `hipModuleLaunchKernel`, `hipExtLaunchKernel`,
|
||||
cooperative launches, `hipLaunchHostFunc`, and libraries that launch through them (rocBLAS,
|
||||
hipBLASLt, MIOpen, hipRTC) would hand HIP a queue it does not know. Header-only device code
|
||||
(Composable Kernel templates, rocPRIM primitives) compiled into your `.hip` works. `hipEventRecord`
|
||||
and `hipStreamWaitEvent` on an engine stream are refused by name.
|
||||
|
||||
The engine also reads the code object, which limits a kernel to: **no hostcall** (`printf`/`assert`
|
||||
are refused at the first launch), **no dynamic stack**, **≤ 768 bytes of kernel arguments**, **≤ 64
|
||||
KiB static+dynamic LDS per dispatch**, and an **uncompressed fat binary**. Device memory a kernel
|
||||
needs comes from the `scratch` hook, never `hipMalloc` — the engine's allocator knows what is left.
|
||||
|
||||
`rad_plugin.h`'s conversions define the result: the reference contract is f32 accumulation, IEEE
|
||||
round-to-nearest-even, **no FMA and no reassociation**. A plugin whose bf16 narrowing truncates
|
||||
cannot be checked against the reference. `rad_load_f32`/`rad_store_f32`/`rad_load_bits`/
|
||||
`rad_store_bits` read and write any core dtype from a plane.
|
||||
|
||||
## Colours of agreement
|
||||
|
||||
- `rad_kernel_concurrent(i)` — kernel `i` touches only its operands and scratch, so independent
|
||||
launches may overlap.
|
||||
- `rad_kernel_collective(i)` — it exchanges data with other ranks; the engine keeps bulk link
|
||||
traffic away. `2` promises more: the launch ends the hold itself once the other ranks' bytes have
|
||||
landed, through a `link_gate` word it reaches at system scope; returning `RAD_LINK_ENDED` says it
|
||||
did.
|
||||
- `rad_kernel_chain(i)` — the kernel can wait in its own prologue instead of behind the packet
|
||||
processor's barrier bit. Chained forms are recognised by `r4dchain` in the name and use a
|
||||
`RAD_CHAIN_REC_BYTES` record at `RAD_CHAIN_KARG_OFF`.
|
||||
- `rad_plugin_reference` / `rad_kernel_reference` — an oracle library (or an oracle row beside a
|
||||
device row). A reference library ranks below every unnamed library, is refused for serving unless
|
||||
`--debug-accept-reference-kernels`, and is what `rad-kbench` checks against.
|
||||
|
||||
## Defining a new op
|
||||
|
||||
Only when `docs/OPS.md` has no name for it. The vocabulary document owns the rules: what a schema
|
||||
must declare, and that **the first plugin in hierarchy order to declare an op fixes its schema** — a
|
||||
later plugin declaring it differently is refused at load, both named. `ctest -R ops_doc` checks the
|
||||
document against the registries, so the rules live there only. An op in `docs/OPS.md` is describable
|
||||
by `libref`, and therefore checkable, even when your kernel cannot describe itself; an op you invent
|
||||
runs only on a library that implements it.
|
||||
|
||||
A schema is positional data, not code:
|
||||
|
||||
```c
|
||||
static const RadParamSpec params[] = {
|
||||
{"M", RAD_P_INT, RAD_REQUIRED, RAD_PROLE_NONE}, /* the ranged extent */
|
||||
{"n", RAD_P_INT, RAD_REQUIRED, RAD_PROLE_NONE},
|
||||
{"eps", RAD_P_F64, RAD_REQUIRED, RAD_PROLE_NONE},
|
||||
{"dtype", RAD_P_STR, RAD_REQUIRED, RAD_PROLE_NONE},
|
||||
};
|
||||
static const RadOperandSpec opds[] = {
|
||||
{"x", RAD_OPD_IN, 0}, {"w", RAD_OPD_WEIGHT, 1}, {"y", RAD_OPD_OUT, 0},
|
||||
};
|
||||
static const RadOpSchema schemas[] = {
|
||||
{"my_op", params, 4, opds, 3, "what it computes, for a human"},
|
||||
};
|
||||
```
|
||||
|
||||
`rad_kernel_schema_count/at` return this table. `RAD_REQUIRED`/`RAD_OPTIONAL`/`RAD_DERIVED` and the
|
||||
operand roles (`IN`, `OUT`, `INOUT`, `WEIGHT`, `WTAB`) are the whole vocabulary; the kernel's
|
||||
constraints and launch do the rest.
|
||||
|
||||
## Quantiser plugins
|
||||
|
||||
`abi/rad_quant.h` is the whole contract:
|
||||
|
||||
```c
|
||||
int rad_quant_count(void);
|
||||
const RadQuantizerInfo* rad_quant_at(int i);
|
||||
|
||||
typedef struct RadQuantizerInfo {
|
||||
const char* name; /* what a recipe names */
|
||||
const char* doc;
|
||||
const RadQuantOption* options; /* key, RAD_P_INT|F64|STR, default text, doc */
|
||||
int n_options;
|
||||
RadQuantEncodingFn encoding; /* the encoding this quantiser would write */
|
||||
RadQuantRowBlockFn row_block; /* rows per quantize() call; 0 = whole weight */
|
||||
RadQuantizeFn quantize; /* logical rows -> planes */
|
||||
RadQuantDecodeFn decode; /* optional, only for a scheme of its own */
|
||||
} RadQuantizerInfo;
|
||||
```
|
||||
|
||||
The flow for each weight a recipe rule picks: ask `encoding` (given name, shape, and — at declare —
|
||||
no movement group), let the architecture declare against that encoding, then hand the weight over in
|
||||
row blocks, so a 51 GB table streams. Other calibration data is the quantiser's own option
|
||||
(`calib=DIR`); the output is **canonical planes only**, so every kernel that accepts the encoding and
|
||||
the core's decoder can read it — that is how the converter measures error without knowing the method.
|
||||
|
||||
A quantiser's own scheme must be decoded by its `decode` hook (the oracle, `rad-kbench` and the
|
||||
error report call it), and must return `RAD_E_UNSUPPORTED` for an encoding it did not write. Serving a
|
||||
new encoding also needs a kernel library whose `layout` accepts it and an architecture that declares
|
||||
the weight at it — a new format is a new kernel row plus a plugin, never a converter-only change.
|
||||
|
||||
Two plugins naming the same quantiser are refused, both named.
|
||||
|
||||
### The encoding model (`abi/rad_encoding.h`)
|
||||
|
||||
An encoding is the code type, an optional codebook, the grouping of scales/zero points/minimums, and
|
||||
a transform. It is **data, not an enum**, so a plugin can invent one. The core can size, slice and
|
||||
place any encoding from the struct alone, and decodes the `plain` and `affine` schemes itself:
|
||||
|
||||
```
|
||||
q = code codes TILED, block [1x1], or [1xV] with a grid
|
||||
q = grid[code][k] grid TABLE [entries x V]
|
||||
q = table[code] table TABLE [1 x entries]
|
||||
q = -q if the sign bit is set signs TILED u1
|
||||
q = q - zero zero TILED
|
||||
v = q * scale * scale.1 * ... every scale level
|
||||
v = v - min * min.1 * ... a minimum in the value's domain
|
||||
w = v . T^-1 the transform, per row over columns
|
||||
```
|
||||
|
||||
So NVFP4 is fp4 codes, e4m3 scale per 16, f32 `scale.1` per tensor; Q4_K is u4 codes, u6 scale and
|
||||
u6 min per 32 each times an f16 per 256; IQ2_XXS is u8 codes per 8 into a 256×8 grid, a sign bit per
|
||||
value and two scale levels.
|
||||
|
||||
- Planes are **TILED** (cover the weight, element `(i,j)` covering a `[br,bc]` block; `0` extent means
|
||||
the whole extent) or **TABLE** (a codebook, grid, rotation parameter; every slice carries it).
|
||||
- Sub-byte elements are a bit stream along the row, lowest bits first; every row starts on a byte.
|
||||
- Transforms: `""`, `fwhtN` (Hadamard before quantisation; the `1/N` lives in the stored scale),
|
||||
`perm` (GPTQ act-order, a TABLE `t.perm`), `givens` (ParoQuant rotations: `t.pairs`, `t.angle`,
|
||||
optional `t.scale`, per-part `t.part`, composed `t.dense`).
|
||||
|
||||
A kernel only *rearranges* planes (`relayout`); choosing numbers is the quantiser's job. That split
|
||||
is why a container never has to be reconverted when a kernel library changes its layout.
|
||||
|
||||
## Build, install, search path
|
||||
|
||||
```cmake
|
||||
cmake_minimum_required(VERSION 3.21)
|
||||
project(mykernels LANGUAGES CXX HIP) # CXX alone for a host-only plugin
|
||||
find_package(radiance REQUIRED)
|
||||
rad_plugin_defaults() # C++20, optimising build unless asked otherwise
|
||||
rad_add_plugin(mykernels kernels SOURCES rows.cpp HIP_SOURCES kernels.hip)
|
||||
```
|
||||
|
||||
The kind is `kernels`, `architectures` or `quantizers` — the directory the engine loads it from.
|
||||
Only `HIP_SOURCES` compile as HIP: for `RAD_GPU_TARGETS` or a `GPU_TARGETS` list, with `-DNDEBUG`
|
||||
outside Debug, uncompressed, against the HIP runtime the installed radiance found. `find_package`
|
||||
also sets `RADIANCE_ABI_VERSION` (to fail at configure time rather than at `dlopen`),
|
||||
`RADIANCE_HOME`, `RADIANCE_KERNELS_DIR`, `RADIANCE_ARCHITECTURES_DIR`,
|
||||
`RADIANCE_QUANTIZERS_DIR`. `cmake --install` puts the plugin under the home of the radiance you built
|
||||
against; `RAD_PLUGIN_INSTALL_HOME` overrides that.
|
||||
|
||||
**`$RADIANCE_HOME` is a search path**: homes separated by `:`, the first winning; each may hold
|
||||
`kernels/`, `architectures/`, `quantizers/`, and an earlier home's same-name file shadows a later
|
||||
one. Your plugins go ahead of an installation without copying it:
|
||||
|
||||
```sh
|
||||
RADIANCE_HOME=$HOME/radiance-plugins:/opt/radiance/share/radiance radiance --model m.rad
|
||||
# in the container:
|
||||
-v /my/plugins:/plugins -e RADIANCE_HOME=/plugins:/opt/radiance/share/radiance
|
||||
```
|
||||
|
||||
The **kernel hierarchy** is `--kernels` (or `$RADIANCE_KERNELS`): `--kernels mykernels,libr4d` puts
|
||||
yours before libr4d. An entry matches a plugin's name or its file (`libr4d`, `r4d`, `libr4d.so`, a
|
||||
path). An entry matching nothing is warned about by name. With no list, every library found loads,
|
||||
reference last.
|
||||
|
||||
## A library for a new card
|
||||
|
||||
The engine's own device code is compiled for every AMD family's generic target
|
||||
(`gfx9-generic`, `gfx9-4-generic`, `gfx10-1-generic`, `gfx10-3-generic`, `gfx11-generic`,
|
||||
`gfx12-generic`) beside the build's targets, so one engine binary starts anywhere. What it computes
|
||||
there depends on the installed kernel library.
|
||||
|
||||
- Configure with `-DRAD_GPU_TARGETS=gfx942` (or `gfx11-generic`). At load the engine asks the runtime
|
||||
which targets the card accepts, in order (chip, then family: `gfx1036`, then `gfx10-3-generic`),
|
||||
with the card's modes (`gfx942:sramecc+:xnack-`), and takes the best fit in the fat binary.
|
||||
- **A library with nothing for this card is left out, not failed**: its device rows leave selection,
|
||||
its host rows stay, and the log says what it carries versus what the card runs. One installation
|
||||
can carry a library per card. `rad_plugin_open` may decline explicitly with `RAD_E_UNSUPPORTED`.
|
||||
- The engine uses discrete cards if any exist (all of the first one's architecture), integrated only
|
||||
when there is nothing else; `ROCR_VISIBLE_DEVICES` chooses explicitly. On integrated graphics,
|
||||
device memory is host memory and the budget leaves an eighth of host RAM to the host.
|
||||
- **fp8 bytes are OCP E4M3** (`e4m3fn`, bias 7, max 448) wherever the engine reads them: container
|
||||
weights, an fp8 KV cache, activations. A card whose fp8 is `e4m3fnuz` (gfx942) converts in its
|
||||
`relayout` hook (the codes keep their bits except `0x80` and decode to half the value, so the scale
|
||||
doubles) and keeps activations/KV in OCP bytes, or converts at its own boundaries.
|
||||
- **Tensor parallelism is ops you supply.** `all_reduce`, `all_gather` and the fused collectives are
|
||||
rows like any other; the engine inserts none and assumes no link.
|
||||
|
||||
## Checking your work
|
||||
|
||||
```sh
|
||||
rad-kbench --kernels mylib,libr4d,libref --report mylib.md --bench
|
||||
```
|
||||
|
||||
- It replays a recorded fixture of `libref`'s answers over real model geometries
|
||||
(`$RADIANCE_HOME/kernels.rkb`, plus `kernels_moe.rkb`), running every library with a row for a
|
||||
case. Correctness uses sampled values **and** the full-tensor norm **and** seeded projections;
|
||||
completeness is the op-by-library matrix; speed is timed on the same bytes.
|
||||
- Red zones: every device allocation carries a poisoned 256-byte margin, verified after the launch.
|
||||
- Fusion: a row with `replaces` is run against its named chain, out of your library, compared with
|
||||
`memcmp`.
|
||||
- A skip is named, not failed — usually a missing `opd_shape`. `--strict` also fails a stale fixture.
|
||||
- `rad-avx-check --ref libref.so --avx mylib.so` checks a **host** library without a card, op by op,
|
||||
at every ISA level and with parallel regions forced onto several threads (for a library exporting
|
||||
`avx_isa_force`/`avx_par_force`).
|
||||
- `rad-schemas mylib.so libref.so` prints every schema disagreement at once (it exits 1 on any), and
|
||||
works on a plugin the engine cannot load.
|
||||
- `--debug-selection` prints which row took each band and which constraint refused each other
|
||||
candidate; `/graph` on a running server shows the declared graph and which declarations the run
|
||||
phase issued.
|
||||
- `rad-info --plugins` lists every plugin on the search path as the engine loads it: kind, version,
|
||||
declarations, reference status, whether its device code runs on this card, and its file.
|
||||
|
||||
## A worked example to copy
|
||||
|
||||
`plugin/spechub/` is the smallest real kernel plugin in the tree: two ops (`mtp_couple_propose`,
|
||||
`mtp_couple_verify`), each with a device row and a host reference row. Read it in this order:
|
||||
|
||||
- `exports.cpp` — the whole export block: info, ABI, schema count/at, kernel count/at, plus
|
||||
`rad_kernel_reference` classifying device rows 0 and host rows 1 so the reference gate does not
|
||||
refuse the device rows.
|
||||
- `spechub.cpp` — the rows, their constraints and their argument readers.
|
||||
- `spechub.hip` / `ref_spechub.cpp` — the device launch and the host oracle.
|
||||
- `CMakeLists.txt` — the `rad_add_plugin` call, the optional `enable_language(HIP)`, and
|
||||
`-ffp-contract=off` because an FMA in the ordered FP64 mass sums would move a token off the host
|
||||
reference.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Radiance Architecture Plugins]] — the third plugin kind, one level up.
|
||||
- [[Radiance Conversion]] — how a quantiser is driven by a recipe.
|
||||
- [[Radiance Tools]] — `rad-kbench`, `rad-tune`, `rad-schemas`, `rad-avx-check` in full.
|
||||
- [[Radiance Serve Options]] — `--kernels`, `--radiance-home`, the tuning aids.
|
||||
@@ -0,0 +1,240 @@
|
||||
---
|
||||
type: "reference"
|
||||
title: "Radiance Serve Options — every flag and the law behind it"
|
||||
description: "Every radiance serve flag grouped by task, with its default and the law that decides when to move it."
|
||||
tags:
|
||||
- llm
|
||||
- radiance
|
||||
- reference
|
||||
- serving
|
||||
timestamp: 2026-10-30
|
||||
---
|
||||
|
||||
# Radiance serve options
|
||||
|
||||
`radiance --help` prints this table. Every numeric option takes a whole number (not `12k`, not
|
||||
`auto`, unless the option names those); a malformed value exits 2. Upstream `main` at 2026-10-30;
|
||||
the fleet is pinned older — see the pin note on [[Radiance Engine]].
|
||||
|
||||
The one thing to read before any table: **the engine derives the VRAM split; no flag sets either
|
||||
pool.** The startup receipt prints every term before anything is allocated.
|
||||
|
||||
## Model, plugins, container
|
||||
|
||||
| flag | default | meaning |
|
||||
|---|---|---|
|
||||
| `--model PATH` | required | a `.rad` container, a checkpoint served as it stands, or a Hugging Face repo id (`owner/name`) fetched before startup |
|
||||
| `--download-dir DIR` | `$XDG_CACHE_HOME/radiance/hf`, else `~/.cache/radiance/hf` | where a repo id is fetched |
|
||||
| `--radiance-home DIRS` | compiled prefix | plugin homes, `:`-separated, first wins |
|
||||
| `--kernels LIST` | every library found | kernel libraries, `,`/`:` separated, first wins; a library not named is not loaded |
|
||||
| `--served-model-name NAME` | container's name | the id `/v1/models` lists and responses carry; a request may name any model |
|
||||
| `--override-chat-template PATH` | container's | a Jinja template served instead; an unreadable or empty file stops startup |
|
||||
| `--generation-config PATH` | none | sampler defaults from a `generation_config.json` (else `<model>.generation.json`, then the container's `generation.*` metadata) |
|
||||
| `--control-vector off\|on\|disabled` | `disabled` | load the GGUF control vector found for the model and start requests off/on; `disabled` declares no projection and refuses a request that asks |
|
||||
| `--api-key KEY` | none | require `Authorization: Bearer KEY` or `X-Api-Key: KEY`, except `/health`, `/v1/health`, `/ping`, the dashboard |
|
||||
| `--mm-max-patches N` | `auto` (16384 with a vision tower) | patches one encoder pass carries: the largest image, 16×16 px a patch (14×14 on GLM-5.3). `0` serves text only and keeps the tower off the card |
|
||||
|
||||
## Batch shape, context, scheduler
|
||||
|
||||
| flag | default | meaning |
|
||||
|---|---|---|
|
||||
| `--max-num-batched-tokens N` | 2048 | the largest token count one step may carry; on MoE the knob that sets prefill chunk size, and the activation arena's size |
|
||||
| `--max-num-seqs N` | 256 | the most sequences one step may carry; the cache's **demand**, never its funding |
|
||||
| `--max-model-len N` | 0 = model's training context | context bound per request. Longer is a 400, never a truncation. Reserves nothing per request; it is the worst case the pool is sized against |
|
||||
| `--busy-prefill-chunk auto\|off\|N` | `auto` | a prefill chunk's share of a step that has other work: `auto` leaves decode rows and waiting requests room, `off` takes the whole step, `N` caps it |
|
||||
| `--prefill-interleave N` | 0 (off) | decode-only steps after each chunk taken beside decoders, at most 64; buys decoder cadence with the prompt's TTFT |
|
||||
| `--prefill-span N` | 1 | checkpoint intervals a prefill **alone in its step** may carry, from its second interval; needs `--max-num-batched-tokens` of N intervals. Every other step is planned at the shared budget; checkpoints inside a span are not written |
|
||||
| `--preempt-wait-ms N` | 5000 | how long a preempted prefill waits for the whole-recompute gate before reclaiming blocks and preempting lower-priority work; `0` falls back next step |
|
||||
| `--rope-scaling SPEC` | the checkpoint's | `yarn:FACTOR[:ORIG]` writes static YaRN over the container's own rotary keys with no reconvert; `none` takes a checkpoint's YaRN off |
|
||||
| `--max-queued-requests N` | 8 × `--max-num-seqs` | admitted, unfinished requests; past it the answer is 429 |
|
||||
| `--retry-after S` | 1 | the `Retry-After` a 429 carries |
|
||||
|
||||
## Memory and placement
|
||||
|
||||
| flag | default | meaning |
|
||||
|---|---|---|
|
||||
| `--placement MODE` | `auto` | `all_vram`, `layer_offload`, `expert_tiered`, `auto` |
|
||||
| `--gpu-headroom-mib N` | 96 | VRAM left unclaimed on every card; too little makes the first launch of a never-run kernel fail inside HIP |
|
||||
| `--host-pool-mib N` | 0 | pinned host tier for weights, **per rank**; also the KV lever under expert tiering (below) |
|
||||
| `--host-headroom-mib N` | 4096 | host RAM left free when startup pins memory; a snapshot store's floor takes the shortfall back only down to 1 GiB |
|
||||
| `--kv-cache-dtype T` | `bf16` | `bf16`, `fp8`, and on Qwen3.8-27B/-Flash-Next `fp8_v4` (V int4) or `k6_v4` (K int6 + V int4). A real quantisation; GLM-5.3 is bf16-only |
|
||||
| `--state-cache-dtype T` | `f32` | `f32` or `f16`: the recurrent delta-net state cache; f16 halves state pool, checkpoints and traffic and is lossy — `qwen35` plugins only |
|
||||
| `--attn-prefill-fp8 MODE` | `off` | `off`, `qk`, `pv`, `both`: prefill attention products in E4M3 over an `fp8`/`fp8_v4`/`k6_v4` cache; lossy, Qwen3.8-27B only where kernels exist |
|
||||
| `--kv-resident N` | 0 (off) | stream the sparse attention group's KV from pinned host RAM, keeping an N-token ring on the card. Raised to 32768 and to one decode step's selection; ignored with a warning where unsupported |
|
||||
| `--embedding-placement ram\|vram` | `ram` | token embedding pinned in host memory and read in place, or on the card. A tied head or a host that cannot pin keeps it on the card; an explicit `ram` refuses instead |
|
||||
| `--vision-tower-placement vram\|ram` | `vram` | encoder towers on the card, or pinned in host memory and read once per image, giving their VRAM to the KV cache |
|
||||
| `--ngram-placement auto\|vram\|ram\|disk` | `auto` | where an n-gram embedding table lives: all of it on every card, a pinned host copy the card reads, or rows from the container through a row cache. `auto` takes the first that fits. `ram=DIR` makes the pinned copy a tmpfs file shared by two engines |
|
||||
| `--weights-disk-tier` | off | read weights that fit neither VRAM nor the host pool from the container with O_DIRECT; every step is paced by the drive |
|
||||
| `--row-prefetch on\|off` | on | ask for a host gather's pages before its kernel; output-neutral |
|
||||
| `--mapped-rewarm on\|off` | on | at startup, re-ask the page cache for a mapped table's pages an earlier run left behind |
|
||||
| `--deterministic` | off | pin placement to the loader's plan so the resident set cannot move under the heat engine; it does **not** make output bit-identical across batch positions |
|
||||
|
||||
## KV cache, host tier, checkpoints
|
||||
|
||||
| flag | default | meaning |
|
||||
|---|---|---|
|
||||
| `--prefix-cache-mib N` | 0 (off) | copy finished conversations' KV to host memory so a return restores instead of re-prefilling; LRU when full; off while a KV group streams |
|
||||
| `--prefix-cache-ttl S` | 3600 | seconds an unused cached conversation is kept; `0` keeps it until memory is wanted |
|
||||
| `--no-prefix-cache` | cache on | disable prefix caching |
|
||||
| `--checkpoint-interval N` | 2048 | linear-attention state checkpoint interval, in tokens; also clamps a hybrid model's prefill chunk |
|
||||
| `--checkpoint-slots N` | `auto` | snapshots held **in total**, in pinned host memory; `auto` = per-policy need for each sequence within an eighth of host RAM; `0` keeps none |
|
||||
| `--checkpoint-policy NAME` | `geometric-backoff` | also `tip-only`, `keep-all`; retention only — writes happen at every interval unless a span skips them |
|
||||
| `--prefix-resume exact\|nearest` | `exact` | `exact` resumes only where a cold prefill starts a chunk, so a hit answers bit-identically; `nearest` takes the deepest snapshot, up to a chunk less re-prefilled |
|
||||
|
||||
## Speculation
|
||||
|
||||
| flag | default | meaning |
|
||||
|---|---|---|
|
||||
| `--num-speculative-tokens N` | `auto` (the drafter's own point; 0 with no head) | speculative window; 0 disables. The ceiling under `adaptive` |
|
||||
| `--spec-ngram K` | 0 (off) | prompt lookup alone, up to K drafts a step from the sequence's own n-grams; leaves a container's draft head out; sets the window to K |
|
||||
| `--spec-ngram-min N` / `--spec-ngram-max N` | 2 / 4 (max 15) | pattern lengths prompt lookup tries, longest first |
|
||||
| `--draft-window fixed\|adaptive` | `fixed` | `adaptive` follows each sequence's acceptance and the measured step cost |
|
||||
| `--draft-short-from N` | `auto` | a block drafter that declares a shorter block runs it from N decoding sequences; `0` never; `auto` = never for a chain, from 2 for a tree |
|
||||
| `--draft-coupling off\|on` | off | sample each proposal from the drafter's own shortlist and couple the target's draw to it; same output law, different tokens for a fixed seed, pays at high temperature |
|
||||
| `--draft-model PATH` | — | **not implemented** at serve: merge the drafter at convert time (`rad-convert --draft-model`) |
|
||||
|
||||
## Tensor parallelism and collectives
|
||||
|
||||
| flag | default | meaning |
|
||||
|---|---|---|
|
||||
| `--tp N` | 1 | tensor-parallel ranks; declare divides dimensions by world size and inserts no collective |
|
||||
| `--tp-wire exact\|wht6` | `exact` | `wht6` is lossy: rotated 6-bit at 2 ranks, tiered int8 over a two-shot at 3–4, unavailable at 8 |
|
||||
| `--tp-wire-min-kb N` | 128 | smallest all-reduce (KiB) the lossy wire compresses; keeps single-sequence decode exact |
|
||||
| `--tp-attn-split heads\|rows` | `heads` | a prefill step's attention split: `rows` gives each rank all heads of its rows plus an all-gather; faster where attention dominates, not bit-identical, plugin-gated |
|
||||
| `--weight-prefetch on\|off` | on | read a GEMM's weight beside the collective before it; `off` when another server shares the cards |
|
||||
| `--p2p auto\|on\|off` | `auto` | `auto` asks the driver; `on` refuses to start without peer access; `off` exchanges through pinned host memory |
|
||||
|
||||
## LoRA adapters
|
||||
|
||||
| flag | default | meaning |
|
||||
|---|---|---|
|
||||
| `--lora NAME=PATH` | none | serve an adapter: a PEFT directory as model NAME (text), or a `.safetensors` a request names in `loras` (image/video). Repeatable |
|
||||
| `--lora-dir DIR` | none | generators: any `NAME.safetensors` under DIR may be requested |
|
||||
| `--max-loras N` | count of `--lora`, 2 with `--lora-dir` alone | adapter slots resident at once |
|
||||
| `--max-lora-rank R` | 16 text / 64 generator | slot rank; a larger text adapter is refused, a larger generator combination truncated |
|
||||
| `--lora-target-modules LIST` | the adapters' own | HF module leaf names adapters may target; needed with `--max-loras` and no `--lora` |
|
||||
|
||||
## Request defaults and bounds
|
||||
|
||||
A request that names a field wins. These set defaults for requests that do not.
|
||||
|
||||
| flag | default | meaning |
|
||||
|---|---|---|
|
||||
| `--temp F`, `--top-k N`, `--top-p F`, `--min-p F` | from the model | sampler defaults; `top_k` 0 is the 1024 most likely candidates |
|
||||
| `--typical-p F` | 1 | typical sampling |
|
||||
| `--presence-penalty F`, `--frequency-penalty F` | 0 | range [-2, 2] |
|
||||
| `--repetition-penalty F`, `--repeat-last-n N` | 1, 64 | `-1` looks back over the whole context |
|
||||
| `--dry-multiplier F`, `--dry-base F`, `--dry-allowed-length N`, `--dry-penalty-last-n N` | 0, 1.75, 2, -1 | DRY; multiplier 0 is off |
|
||||
| `--dry-sequence-breakers JSON` | `["\n", ":", "\"", "*"]` | at most 16 strings |
|
||||
| `--xtc-probability F`, `--xtc-threshold F` | 0, 0.1 | range [0, 1] |
|
||||
| `--default-max-tokens N` | `auto` | `max_tokens` for a request that sends none; `auto` is what the context leaves |
|
||||
| `--max-tokens-cap N` | 0 (none) | a larger request is **cut to it**, not refused; reply ends `finish_reason: "length"` |
|
||||
| `--max-n N` | max(8, `--max-num-seqs`) | the largest fan-out `n` |
|
||||
| `--max-stop-strings N`, `--max-stop-bytes N` | 64, 4096 | stop-string limits |
|
||||
| `--chat-template-kwargs JSON` | none | variables for the template on every chat request; a JSON object or `@FILE`; repeatable |
|
||||
| `--reasoning-effort S` | template's own | default `reasoning_effort`; passed to the template verbatim, so the legal set is the template's |
|
||||
| `--reasoning-effort-map MAP` | none | `FROM=TO,...`, checked against the template at startup; `none`/`minimal` turn thinking off |
|
||||
| `--thinking-token-budget N` | none | default budget: at N reasoning tokens the block is closed and the answer follows |
|
||||
| `--reasoning-format auto\|none` | `auto` | `auto` returns reasoning as `reasoning_content`; `none` leaves it inline |
|
||||
|
||||
## Server and HTTP
|
||||
|
||||
| flag | default | meaning |
|
||||
|---|---|---|
|
||||
| `--host ADDR` | 0.0.0.0 | listen address |
|
||||
| `--port N` | 8000 | listen port |
|
||||
| `--http-threads N` | one per hardware thread, at least 4 | workers kept while idle |
|
||||
| `--read-timeout S` | 30 | a client may go silent this long while sending |
|
||||
| `--write-timeout S` | 600 | one write to a client may block this long |
|
||||
| `--keep-alive-timeout S` | 5 | idle connection hold |
|
||||
| `--max-body-mib N` | 512 | largest body; images and video ride base64 inside it |
|
||||
| `--no-cors` | CORS on | send no CORS headers |
|
||||
|
||||
## Media preprocessing
|
||||
|
||||
Each replaces what the container's preprocessor config states, and does nothing for a model that
|
||||
takes no media.
|
||||
|
||||
| flag | default | meaning |
|
||||
|---|---|---|
|
||||
| `--image-min-pixels N`, `--image-max-pixels N` | container's | the band an image resizes into; one encoder pass still bounds the top |
|
||||
| `--video-min-pixels N`, `--video-max-pixels N` | container's | same, for a video's sampled frames together |
|
||||
| `--video-fps F` | container's | frames sampled per second |
|
||||
| `--video-min-frames N`, `--video-max-frames N` | container's | frame-count bounds |
|
||||
| `--video-max-frame-tokens N` | container's | the most prompt tokens one frame becomes; `0` no cap; a token-budget processor takes 70/140/280/560/1120 |
|
||||
| `--image-tokens N` | container's | a token-budget processor's image budget (Gemma 4's fixed set; GLM-5.3's ceiling ≤ `--mm-max-patches`/4) |
|
||||
| `--max-source-pixels N` | 67108864 | largest input the decoder accepts before resize |
|
||||
|
||||
## Quality measurement (instead of serving)
|
||||
|
||||
| flag | meaning |
|
||||
|---|---|
|
||||
| `--kld-record DIR` | run `--kld-corpus` through prefill and write this model's per-position log-probabilities to DIR: the reference |
|
||||
| `--kld-ref DIR` | score this model against the reference: mean/median/percentile KL and top-1 agreement |
|
||||
| `--kld-corpus FILE` | JSONL, one `{"prompt", "score_from"?, "source"?}` a line; `--kld-record` only |
|
||||
| `--kld-out FILE` | the report as JSON; `FILE.rows` holds every position's numbers |
|
||||
| `--kld-lora NAME`, `--kld-lora-sources PREFIX` | score through a served adapter, all documents or a `source` prefix |
|
||||
|
||||
The mode turns speculation and the prefix cache off and refuses `--num-speculative-tokens > 0`.
|
||||
See [[Radiance Conversion]] for the corpus and the scoring discipline.
|
||||
|
||||
## Diagnostics
|
||||
|
||||
| flag | meaning |
|
||||
|---|---|
|
||||
| `--debug-graph` | print the declared graph, every op's resolved kernel and why, then exit |
|
||||
| `--debug-selection` | the selection table only |
|
||||
| `--debug-placement` | the placement plan only |
|
||||
| `--debug-tape-dump` | the first recorded pass of each kind (`TAPEDUMP` on stderr, every rank) |
|
||||
| `--debug-draft-oracle FILE` | verify-path test: prompt lookup proposes from FILE with 30 % corrupted drafts; the text must not move |
|
||||
| `--debug-accept-reference-kernels` | serve even with `libref` rows — not a production configuration |
|
||||
| `--profile-ops` | per-op device timing |
|
||||
| `--expert-stats FILE` | routing counts per layer and step kind, JSON every 30 s and at shutdown |
|
||||
| `--expert-profile FILE` | rank the expert plane by a stats file's `decode` counts instead of the container profile |
|
||||
| `--expert-pin-top F` | pin the hottest fraction F of routed experts so heat never demotes them |
|
||||
| `--draft-expert-placement host\|auto` | pin a draft head's experts in the host pool before any strategy runs |
|
||||
| `--live` | the terminal live view; redirected, one `key=value` line |
|
||||
| `-v`, `-vv` | verbose, very verbose |
|
||||
|
||||
## The laws behind the knobs
|
||||
|
||||
**The split is derived, and the receipt is the referee.** Non-expert weights come off the top of
|
||||
the card; the remainder (`elastic`) is shared by the expert slab and the KV cache. `--max-model-len`
|
||||
× `--max-num-seqs` is the cache's **demand**; the funding is `elastic − expert_floor`. Under a
|
||||
lending cache (`auto`/`expert_tiered`, no `--deterministic`, VMM present) the cache takes what it
|
||||
can address and lends unused blocks back to experts. **Every MiB added to `--host-pool-mib` lowers
|
||||
the expert floor one-for-one and the cache takes it** — until the cache already holds its whole
|
||||
ceiling, where the extra host room shows up as unclaimed. Read the boot line
|
||||
`kv: the pool backs N % of the worst case …`, never arithmetic.
|
||||
|
||||
**The activation arena comes off the budget before the split**, so both sides pay for
|
||||
`--max-num-batched-tokens`. While steps are small most of it is lent back to the expert slab; a
|
||||
larger step reclaims it synchronously. That is why a bigger chunk can cost decode residency. On a
|
||||
hybrid model the chunk is clamped to `--checkpoint-interval` before the token budget, and startup
|
||||
prints which bound applies.
|
||||
|
||||
**The KV dtype law.** `bf16` and `fp8` are engine-wide; `fp8_v4`/`k6_v4` are narrower layouts whose
|
||||
kernels exist only for particular attention geometries, and the server refuses them elsewhere by
|
||||
plugin name. A drafter's cache stays fp8. Recurrent state and the QSA indexer keep their own widths.
|
||||
|
||||
**The slot law.** A slot costs recurrent state, block tables at full context, snapshot-store RAM
|
||||
and all-reduce rows — never speed by itself. Raising `--max-num-seqs` raises demand; whether it is
|
||||
funded is the resolver's answer.
|
||||
|
||||
**The spec law.** Measure speculation by step time, not tok/s. Prompt lookup pays where the reply
|
||||
copies its context and costs a verify row elsewhere. `adaptive` moves the depth only after eight
|
||||
steps and a 2 % gain, because a new window is a new pass shape.
|
||||
|
||||
**Batch shape is part of the configuration.** A different chunk, span or spec depth changes the
|
||||
summation order, so greedy text can differ at a tie. Reproducibility holds per configuration, never
|
||||
across one.
|
||||
|
||||
**Unknown request fields are ignored; known-but-unimplemented fields are guarded** (accepted at
|
||||
their off value, refused otherwise) — the list is on [[Radiance HTTP API]].
|
||||
|
||||
## Related
|
||||
|
||||
- [[Radiance HTTP API]] — what the flags expose on the wire.
|
||||
- [[Radiance Operations]] — the container contract and readiness.
|
||||
- [[Radiance Plugins]] — how a flag reaches a plugin (`RadBuildCtx`).
|
||||
- [[frank-archon]] — the measured deployment of this resolver on two R9700s.
|
||||
@@ -0,0 +1,322 @@
|
||||
---
|
||||
type: "reference"
|
||||
title: "Radiance Tools — inspect, check, tune, tokenise"
|
||||
description: "The radiance inspection, measurement and tokeniser tools, with the flags and verdicts each one reports."
|
||||
tags:
|
||||
- llm
|
||||
- radiance
|
||||
- tools
|
||||
- reference
|
||||
timestamp: 2026-10-30
|
||||
---
|
||||
|
||||
# Radiance tools
|
||||
|
||||
Six binaries and `rad-export` are installed beside `radiance`; `rad-avx-check`, `r4d_selftest` and
|
||||
`rad-chatparse` live in the build tree. Every tool that loads plugins honours `RADIANCE_HOME`,
|
||||
`RADIANCE_KERNELS` and `--kernels` as the engine does. A numeric option takes a whole number: `12k`,
|
||||
`auto` (where the option does not name it) or an empty value exits 2.
|
||||
|
||||
`rad-convert`, `rad-kbench` and `rad-tune` run the engine's startup with a different last step: the
|
||||
same plugins, in the same hierarchy, through the same declare phase, so they resolve kernels exactly
|
||||
as a serve would.
|
||||
|
||||
## `rad-info` — what a container holds
|
||||
|
||||
```
|
||||
rad-info [options] <model.rad>
|
||||
rad-info --plugins [--home DIRS]
|
||||
|
||||
-v, --verbose list every tensor, not just the per-unit summary
|
||||
--meta metadata table, then stop
|
||||
--vocab vocab summary and normaliser chain, then stop
|
||||
--profile N top N expert-profile entries (default 20, 0 = all)
|
||||
--recipe the recipe the container was converted with, then stop
|
||||
--plugins every plugin the search path offers, as the engine loads it
|
||||
--home DIRS search path for --plugins (default $RADIANCE_HOME)
|
||||
```
|
||||
|
||||
It opens the container read-only and loads no plugins (except for `--plugins`): a container names no
|
||||
kernel library. Output is aligned; byte counts are binary units and each count says what it counts.
|
||||
|
||||
```sh
|
||||
rad-info m.rad # format, architecture, model, source quant, recipe, sections
|
||||
rad-info --recipe m.rad # the recipe, expanded, one rule a line
|
||||
rad-info -v m.rad | head -40 # every tensor, encoding, shape and what made it
|
||||
rad-info --plugins # kind, version, declarations, reference, device fit, file
|
||||
```
|
||||
|
||||
The section table (strings, metadata, directory, planes, encodings, vocab, profile, data) is the
|
||||
container's layout; the metadata table, the weight directory grouped by layer and movement unit, the
|
||||
vocab, and the expert profile follow.
|
||||
|
||||
## `rad-kbench` — correctness, completeness, speed
|
||||
|
||||
```
|
||||
rad-kbench [options] replay the recorded fixture (no model)
|
||||
rad-kbench -m <model.rad> [options] check against a real container
|
||||
rad-kbench -m <model.rad> --record FILE record a new fixture
|
||||
rad-kbench -m <model.rad> --skeleton FILE write a metadata-only copy to record from
|
||||
```
|
||||
|
||||
**Replay** needs no model, weights or card. Geometries, seeds and `libref`'s reduced answers are
|
||||
checked in as `data/kernels.rkb` (dense model, block drafter, gqa 6) and `data/kernels_moe.rkb`
|
||||
(routed MoE, MTP head, gqa 8). `--fixture` is repeatable; `ctest` runs three replays — the other
|
||||
libraries against each recording, and `libref` against both at tolerance zero (its regression test,
|
||||
since a live comparison moves both sides together).
|
||||
|
||||
Each library is asked for its row as the engine would ask it with that library alone: same
|
||||
constraints, derived parameters and weight-encoding filter, so a row the engine would pass over does
|
||||
not run here either (`--all-rows` runs every row whose constraints hold).
|
||||
|
||||
**Against a container** (`-m`), weight operands are read from it: each weight's canonical planes,
|
||||
which the kernel gets through its own `layout`/`relayout` hooks and the oracle gets decoded, and
|
||||
every expert of a weight table. This is the only test of the artifact and of the layout hooks.
|
||||
|
||||
**Recording** takes the reference's answer once at every point of every declared op, whichever
|
||||
library resolved it. Weights are drawn from a seed, not read from the container, so any machine can
|
||||
reproduce the reference and no card is needed. A skeleton (tables intact, data a hole) records the
|
||||
same fixture in a few megabytes.
|
||||
|
||||
```sh
|
||||
rad-kbench -m model.rad --skeleton model.skel.rad
|
||||
rad-kbench -m model.skel.rad --max-cases 1 --record data/kernels.rkb
|
||||
rad-kbench -m moe.rad --max-cases 1 --record data/kernels_moe.rkb
|
||||
```
|
||||
|
||||
### Verdicts
|
||||
|
||||
- An output is compared on sampled positions, the full-tensor norm, and 32 seeded random-sign
|
||||
projections (which estimate rel_l2 to about 25 %). The largest of the three is the error.
|
||||
- Tolerances are per op and activation dtype (`dtype`, else `q_dtype`, else `kv_dtype`). Copies,
|
||||
gathers, scatters and cache stores are exact. Every op in the fixtures has a row of its own.
|
||||
- Ids are compared exactly; where the contract leaves a tie open (`sample_argmax`, `router_topk`,
|
||||
`sample_topk`), an id indexing a value equal to the reference id's is accepted.
|
||||
- A case whose reference and output are both identically zero checked nothing: **vacuous**, not
|
||||
passed.
|
||||
- A failure, a red-zone overrun, a fusion that differs from its chain, or a host kernel that wrote
|
||||
into an input exits 1. `--strict` also fails a stale fixture.
|
||||
- `--coverage DIR` keeps one file per fixture/library of the rows a replay reaches
|
||||
(`data/kbench_rows/`); a listed row not reached fails, unless a memory skip kept its cases out.
|
||||
- Every pass ends with a line saying where its time went (draw, hashing, layout, upload, launch,
|
||||
read-back, comparison, cache, timing).
|
||||
|
||||
### Timing
|
||||
|
||||
`--bench` times every case that passes, on the bytes it was checked on.
|
||||
|
||||
- Device cases are timed **after** the checks, operands still on the device, after one sustained
|
||||
warm-up. A time is device timestamps around a batch of back-to-back launches divided by the launch
|
||||
count; the batch is queued behind a gate the host opens once submission is done, so neither the
|
||||
host rate nor a sync is in the interval. Two untimed launches precede every batch; batches repeat
|
||||
until the median's 95 % interval is within 1 % or `--bench-reps` is reached.
|
||||
- **Cold by rotation**: weights and inputs of 1/32 of the footprint or more are read from copies
|
||||
spanning the footprint (`--bench-flush`, default 256 MiB, at least twice an R9700's caches), so
|
||||
each launch reads them from memory as streamed weights do. Smaller inputs stay warm, as the
|
||||
previous op leaves them.
|
||||
- In-place operands are restored from a pristine copy before every launch; the restores are timed
|
||||
alone and subtracted.
|
||||
- Timing draws expert offsets/routing skewed (Zipf 0.5) and block tables as permutations; the
|
||||
checked launch keeps the recorded draw.
|
||||
- The header prints the calibration: launch floor and streaming-read bandwidth warm and cold. If a
|
||||
smaller cold probe runs >1.5× the largest's rate the rotation was cache-served and device times
|
||||
are withheld. Throttler bits that came on during a batch repeat it, up to three times.
|
||||
- Host cases are timed in their own column, after an untimed launch, weights rotated over at least
|
||||
256 MiB (twice LLC), for at most two seconds a case.
|
||||
|
||||
### Flags
|
||||
|
||||
| flag | meaning |
|
||||
|---|---|
|
||||
| `--fixture FILE` | replay recording (default `$RADIANCE_HOME/kernels.rkb`); repeatable |
|
||||
| `--record FILE` | with `-m`: write a fixture |
|
||||
| `--skeleton FILE` | with `-m`: write a sparse container copy |
|
||||
| `--rerecord FILE` | replay: write the fixtures' cases again with the reference's answers |
|
||||
| `--record-rows FILE` | replay: record a row-driven fixture from the cases |
|
||||
| `-m, --model FILE` | the `.rad` to read weight operands from |
|
||||
| `--home DIR`, `--kernels A:B` | plugin home; kernel libraries (default every one found) |
|
||||
| `--op NAME` | only this op (repeatable prefix match) |
|
||||
| `--ref NAME` | the oracle plugin's name (default: the library declaring itself a reference) |
|
||||
| `--no-ref` | replay without the reference among the tested libraries |
|
||||
| `--strict` | replay: a stale fixture fails |
|
||||
| `--all-rows` | run every row whose constraints hold |
|
||||
| `--coverage DIR`, `--coverage-write` | the row-coverage ratchet |
|
||||
| `--no-cache`, `--cache-dir DIR` | run the reference instead of reusing cached answers; cache location |
|
||||
| `--seed N` | base seed; every case derives its own |
|
||||
| `--tol X` | override every tolerance (bisecting only) |
|
||||
| `--max-ctx N` | deepest context considered (declare and the attention sweep) |
|
||||
| `--max-tok N` | `max_tok` declare runs at (default 512, small on purpose) |
|
||||
| `--mm-max-patches N` | encoder pass declare runs at (default 2048 when a tower is present; 0 leaves it out) |
|
||||
| `--kv-cache-dtype T` | `bf16` or `fp8`, as on the engine |
|
||||
| `--num-speculative-tokens N` | draft depth to declare at (default: the container's) |
|
||||
| `--max-cases N` | runs per distinct case (default 4; 0 = every one) |
|
||||
| `--max-bytes MIB` | skip a case needing more than this, by name (default: half available memory, 512–8192) |
|
||||
| `--no-fusion` | skip the fusion oracle |
|
||||
| `--bench`, `--bench-iters N`, `--bench-reps N`, `--bench-warm-ms N`, `--bench-flush MIB` | timing |
|
||||
| `--timing-only` | time every device kernel, check nothing (refuses `--record`) |
|
||||
| `--report FILE`, `--analysis FILE` | markdown report; splice in `tools/kbench-analysis.sh` output |
|
||||
| `-v` | every case, not only failures |
|
||||
|
||||
Notes: `--max-bytes` matters on a loaded machine — one case holds each operand as drawn, as the
|
||||
kernel's copy, and on the device; without the cap a small machine is OOM-killed with no indication of
|
||||
the op. `--bench` times cold on purpose: an R9700's 52 MB bf16 weight read warm comes from cache at
|
||||
1.45 TB/s against 565 GB/s from memory, so warm times rank shapes but do not size a roofline. A
|
||||
kernel that cannot lay out an encoding is a **skip**, not a failure (`RAD_E_DTYPE`), and the report
|
||||
names the kernel, operand and encoding.
|
||||
|
||||
## `rad-tune` — measure the instantiation space
|
||||
|
||||
```
|
||||
rad-tune -m <model.rad> [options]
|
||||
|
||||
-m, --model FILE the .rad whose declared graph supplies the shapes
|
||||
--home DIR $RADIANCE_HOME; cache at <home>/tune/<machine>.tune
|
||||
--kernels A:B kernel libraries to load, first wins
|
||||
--op NAME only ops whose name starts with NAME
|
||||
--machine KEY override the machine key (writing a cache for another machine)
|
||||
--reps N timing reps, median reported (default 5)
|
||||
--iters N launches per rep (default 32)
|
||||
--max-tok N max_tok declare runs at (default 8192)
|
||||
--tp N tensor-parallel width to declare at (default 1)
|
||||
--num-speculative-tokens N|auto draft depth to declare at
|
||||
--dry-run list what would be measured
|
||||
-v, --verbose every candidate, not just the winner
|
||||
```
|
||||
|
||||
Kernels declare tunable axes, not named variants; `rad-tune` walks the cross product at this model's
|
||||
shapes and writes the winner per (op, geometry) to the cache the selector reads at startup.
|
||||
|
||||
- **Declare at the shapes you serve**: `--tp N` divides every sharded N; a declare at world size 1
|
||||
measures a program no `--tp 2` serve runs. `--num-speculative-tokens` puts the drafter's kernels in
|
||||
the measured graph.
|
||||
- At `--tp > 1` every rank declares at once, each on its own thread and device: a collective's
|
||||
`init` is a rendezvous, so declaring rank 0 alone hangs.
|
||||
- A collective is measured as the exchange: per-rank operands (own seeds, so identical halves cannot
|
||||
pass), own scratch and stream, own instance; every rank fires at once and the clock stops when the
|
||||
last stream drains. Non-collective rows run on rank 0 alone.
|
||||
- `RADIANCE_TUNE` pins a kernel's axes to compare a tuned cache against the defaults — a measurement
|
||||
aid, not a deployment setting.
|
||||
|
||||
## `rad-schemas` — the op vocabulary diff
|
||||
|
||||
```
|
||||
rad-schemas libref.so print one plugin's vocabulary
|
||||
rad-schemas libr4d.so libref.so diff them, in hierarchy order
|
||||
```
|
||||
|
||||
The first plugin in hierarchy order to declare an op fixes its schema; a later plugin declaring the
|
||||
same op differently is refused at load, both named (arguments are positional, so disagreeing schemas
|
||||
would make one call mean different things). `rad-schemas` prints every disagreement at once, so a new
|
||||
plugin can be conformed before it is loaded beside another. It exits 1 on any disagreement. It links
|
||||
no core — it `dlopen`s plugins and reads static tables, so it works on a plugin the engine cannot
|
||||
load.
|
||||
|
||||
## `rad-avx-check` — host libraries without a card
|
||||
|
||||
```
|
||||
rad-avx-check --ref libref.so --avx LIB.so [options]
|
||||
|
||||
--ref PATH the oracle plugin (default libref.so)
|
||||
--avx PATH the plugin under test (default libavx.so)
|
||||
--op NAME only this op
|
||||
--level N only this ISA level (0 scalar, 1 avx, 2 avx2, 3 avx512)
|
||||
--threads N rerun every case with each parallel region on N threads (default 4; 0 skips)
|
||||
--tol X override every tolerance
|
||||
-v every case
|
||||
```
|
||||
|
||||
Checks a host library op by op against `libref`, on the CPU, with neither engine nor card: two
|
||||
`dlopen`s, a geometry table (`libavx/avx_cases.cpp`), libref's operand descriptions and hand-built
|
||||
cases. For every case: schemas as the loader compares them, outputs within `rad-kbench`'s tolerances,
|
||||
guard bytes, inputs unchanged, and an oracle output that is not just the draw. `libavx` is also
|
||||
checked at every ISA level the CPU executes and with parallel regions forced onto several threads.
|
||||
A new host library is checked with the same command; it is treated as a host library (not `libavx`),
|
||||
so libref ops it has no row for are listed rather than failed, and the ISA/thread sweeps run only if
|
||||
it exports `avx_isa_force`/`avx_par_force`. A row returning `RAD_E_UNSUPPORTED` for a geometry libref
|
||||
serves fails: the selector would resolve to it and the step would fail at issue.
|
||||
|
||||
## `r4d_selftest` — libr4d's own check and timer
|
||||
|
||||
```
|
||||
r4d_selftest <libr4d.so> [--case <row>]
|
||||
r4d_selftest <libr4d.so> --perf N K M [iters] (and the other --perf* modes in its usage)
|
||||
```
|
||||
|
||||
With no mode it runs the catalogue and static checks, then each row's smoke and numeric cases in a
|
||||
child process of its own, with fusion claims compared byte for byte (`ctest` labels `slow`, `gpu`).
|
||||
`--case` runs one row in-process as the reproducer. `--perf` covers the fp8 decode GEMMs and head;
|
||||
`--perfmxd` the dense OCP-MXFP4 rows with the weight bandwidth each reaches; `--perfmx` the MXFP4
|
||||
expert GEMM; `--cold MIB` times each launch alone behind a cache flush. `--perfar8` prints the
|
||||
eight-rank all-reduce shard weights and times `lockstep`/`ring`/`split` forms at every piece count;
|
||||
`R4D_SELFTEST_ONE_CARD=1` runs them on one card for their bits only.
|
||||
|
||||
## `rad-tokcheck` — tokeniser against its references
|
||||
|
||||
```
|
||||
rad-tokcheck [options] <tokenizer.json> <corpus-file>...
|
||||
|
||||
--hf-dir DIR compare against HuggingFace ids in DIR/<corpus basename>.ids
|
||||
(little-endian int32, encode(text, add_special_tokens=False))
|
||||
--no-ref skip the reference encoder (slow)
|
||||
--user encode with parse_special off
|
||||
--turns N turns in the multi-turn measurement (default 40; 0 skips)
|
||||
--repeat N timing repeats, best taken (default 3)
|
||||
--context N ids shown either side of a difference (default 6)
|
||||
```
|
||||
|
||||
Per corpus file: ids match `tools/tokref`'s reference encoder (llama.cpp's splitters and
|
||||
`std::regex`); ids match HuggingFace `tokenizers` read from files made elsewhere; and speed — MB/s
|
||||
for both encoders, plus the cost of re-tokenising a growing conversation with and without the
|
||||
segment cache. Any difference prints the ids and surrounding text and exits nonzero, so it gates a
|
||||
change.
|
||||
|
||||
## `rad-chatparse` — where a tool call goes wrong
|
||||
|
||||
```
|
||||
rad-chatparse --request REQ.json --rad MODEL.rad --reply R.txt
|
||||
rad-chatparse --render-raw T.jinja --context CTX.json
|
||||
```
|
||||
|
||||
Renders one OpenAI chat request as the server would (container template or
|
||||
`--override-chat-template`, `--chat-template-kwargs`, the plugin's declared reply format, or the
|
||||
template's own with `--no-plugin`), then reads each reply file as the model's output: parsed whole,
|
||||
parsed through the SSE stream as a client reassembles it, and run through the request's tool-call
|
||||
grammar. It prints one JSON object; exit 1 when the two parses disagree or the grammar refuses a
|
||||
token. No weights are loaded.
|
||||
|
||||
## The KL mode
|
||||
|
||||
`radiance --kld-record/--kld-ref/--kld-corpus/--kld-out/--kld-lora` is a mode of the engine, not a
|
||||
separate tool: same placement and kernels, prefill instead of listening, then exit. The flags and the
|
||||
corpus format are on [[Radiance Conversion]].
|
||||
|
||||
## Shell harnesses (`scripts/`)
|
||||
|
||||
They drive a running engine: bring a model up, measure it, assert on what came back. Each header
|
||||
records why it measures the way it does.
|
||||
|
||||
| script | what it does |
|
||||
|---|---|
|
||||
| `mcserve.sh`, `fnserve.sh` | bring MiniCPM5-2B+DSpark / Flash-Next up and leave it running (env knobs: `P TP MLEN SEQS M KVD HR MNBT EXTRA …`) |
|
||||
| `fnstop.sh` | stop whichever is running and wait for exit |
|
||||
| `convdspark.sh`, `convfn.sh` | rebuild the two reference containers (`convfn.sh` needs `CALIB=`) |
|
||||
| `hfget.sh` | resumable Hugging Face fetch with a content manifest |
|
||||
| `fnpf.sh`, `fnconc.sh`, `cstep2.sh`, `ident.sh`, `cardclock.py` | attach to a live server: prefill rate; prefill-vs-decode contention; steady-state decode ms/step per concurrency; run-to-run hashes; which clock limit binds and which rank the TP step waits for |
|
||||
| `fndrop.sh` | disconnect a client mid-request and check every other answer against a reference |
|
||||
| `tierident.sh`, `tiercross.sh` | whether a session that left VRAM returns byte for byte (attention and recurrent state), including two sessions in flight |
|
||||
| `mctools.sh`, `mcstream.sh`, `mcagent.sh`, `mcdecl.sh` | tool-call parsing shapes; streamed tool calls on raw SSE; speculation on an agentic workload; whether the drafter proposes |
|
||||
| `dprofd.sh`, `fntrace.sh`, `mctrace.sh`, `oracle.sh` | per-op device timing; rocprofv3 traces; every op against libref on the operands the model issued |
|
||||
| `kv-parity.sh`, `kv-stage-parity-64k.sh` | KV streaming: covering ring and staged prefill are byte-identical to streaming off |
|
||||
|
||||
**Not all of them leave a server alone.** `fnserve.sh`, `mcserve.sh`, `fnstop.sh`, `fntrace.sh`,
|
||||
`dprofd.sh`, `mctrace.sh`, `mcagent.sh` and `mcdecl.sh` kill any running engine (`pkill` on the
|
||||
binary), whatever port it is on; `oracle.sh` stops whatever listens on its own port. `cstep2.sh`,
|
||||
`ident.sh`, `tierident.sh`, `tiercross.sh`, `mctools.sh`, `mcstream.sh`, `hfget.sh`, `convfn.sh`,
|
||||
`convdspark.sh`, `fnpf.sh`, `fnconc.sh`, `fndrop.sh` and `cardclock.py` leave it running
|
||||
(`fndrop.sh` deliberately disconnects clients). Never point the stopping ones at a production port.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Radiance Conversion]] — `rad-convert`, `rad-export`, the KL mode.
|
||||
- [[Radiance Plugins]] — what `rad-kbench`/`rad-tune`/`rad-schemas` check.
|
||||
- [[Radiance Operations]] — the startup receipt these tools corroborate.
|
||||
@@ -683,5 +683,6 @@ vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
|
||||
- Retired-era leftovers in-tree: `scripts/patches/glm53-*.patch`
|
||||
`scripts/patches/gpu_worker-kv-cap.patch` (fork archaeology; none applies byte-exact today).
|
||||
|
||||
Related: [[frank-dragoon]] (the abliterated family), [[frank-zealot]],
|
||||
---
|
||||
Related: [[frank-dragoon]] (the abliterated family), [[frank-zealot]], and the engine reference —
|
||||
[[Radiance Engine]], [[Radiance Serve Options]], [[Radiance Conversion]], [[Radiance Plugins]],
|
||||
[[Radiance Operations]].
|
||||
@@ -13,6 +13,20 @@ the endpoint's **Eras** table, and the numbers a retired era measured are gone.
|
||||
| [[frank-dragoon]] | The iGPU endpoint: live shape, every era with its rollback path, the laws it paid for, and the files that own it. |
|
||||
| [[frank-zealot]] | The RTX 3090 endpoint: live shape exactly as the env ships it, every era with its rollback path, and its card laws. |
|
||||
|
||||
## Radiance engine reference
|
||||
|
||||
| Page | Owns |
|
||||
|---|---|
|
||||
| [[Radiance Engine]] | The radiance inference engine: its shape, plugin model, container format, and how the reference pages fit together. |
|
||||
| [[Radiance Serve Options]] | Every radiance serve flag grouped by task, with its default and the law that decides when to move it. |
|
||||
| [[Radiance HTTP API]] | The HTTP surface radiance serves: endpoints, request fields, streaming, tools, and refusal rules. |
|
||||
| [[Radiance Conversion]] | Turning a checkpoint into a .rad container: rad-convert, the recipe language, quantisers, reuse and export. |
|
||||
| [[Radiance Model Families]] | Which radiance plugin claims which model family, and the conversion command and recipe of record for each. |
|
||||
| [[Radiance Plugins]] | Writing radiance kernel and quantiser plugins: the ABI, kernel rows, hooks, encodings, and the build path. |
|
||||
| [[Radiance Architecture Plugins]] | Writing a radiance architecture plugin: declaring weights, buffers, KV groups and ops, then issuing the step. |
|
||||
| [[Radiance Tools]] | The radiance inspection, measurement and tokeniser tools, with the flags and verdicts each one reports. |
|
||||
| [[Radiance Operations]] | Running a radiance server: the container contract, host prerequisites, readiness, ledgers and safe restarts. |
|
||||
|
||||
## Retired endpoints
|
||||
|
||||
| Page | Owns |
|
||||
|
||||
@@ -162,6 +162,15 @@ is a locator: one row, one clause, no numbers.
|
||||
| [[PCIe Link and Atomics]] | Link probing, the streaming-versus-compute ceiling arithmetic, and the atomics wall that decides multi-GPU collectives. |
|
||||
| [[Quantization by Hardware Architecture]] | This page maps LLM quantization formats to the GPU and CPU architectures that support them. |
|
||||
| [[Quantization Production]] | Making a quantized payload here: the conversion recipe, the scoring gate, and the campaign ladder behind a shipped file. |
|
||||
| [[Radiance Architecture Plugins]] | Writing a radiance architecture plugin: declaring weights, buffers, KV groups and ops, then issuing the step. |
|
||||
| [[Radiance Conversion]] | Turning a checkpoint into a .rad container: rad-convert, the recipe language, quantisers, reuse and export. |
|
||||
| [[Radiance Engine]] | The radiance inference engine: its shape, plugin model, container format, and how the reference pages fit together. |
|
||||
| [[Radiance HTTP API]] | The HTTP surface radiance serves: endpoints, request fields, streaming, tools, and refusal rules. |
|
||||
| [[Radiance Model Families]] | Which radiance plugin claims which model family, and the conversion command and recipe of record for each. |
|
||||
| [[Radiance Operations]] | Running a radiance server: the container contract, host prerequisites, readiness, ledgers and safe restarts. |
|
||||
| [[Radiance Plugins]] | Writing radiance kernel and quantiser plugins: the ABI, kernel rows, hooks, encodings, and the build path. |
|
||||
| [[Radiance Serve Options]] | Every radiance serve flag grouped by task, with its default and the law that decides when to move it. |
|
||||
| [[Radiance Tools]] | The radiance inspection, measurement and tokeniser tools, with the flags and verdicts each one reports. |
|
||||
| [[RDNA4 MoE Expert Offload VRAM Budget]] | Expert-offload arithmetic on the RDNA4 fork: movable terms, the utilisation law, graph charge, ceilings, context ladder. |
|
||||
| [[Shared Memory Architecture Dankclaw]] | Dankclaw uses AMD Strix Halo APU with unified memory architecture. |
|
||||
|
||||
|
||||
Reference in new issue
Block a user