serverAppConfigs/serverAppConfigs-docs/LLM/Radiance Architecture Plugins.md

---
type: "howto"
title: "Radiance Architecture Plugins — declaring a model"
description: "Writing a radiance architecture plugin: declaring weights, buffers, KV groups and ops, then issuing the step."
tags:
  - llm
  - radiance
  - plugins
  - howto
timestamp: 2026-10-30
---

# Radiance architecture plugins

An architecture plugin declares a model in two phases: `rad_arch_declare` names every weight,
buffer, KV group and op **once**, against the model's metadata; `rad_arch_step` issues the ops for
one step. The engine picks the plugin by the exact `(rad_arch_id, rad_arch_quant)` pair — no
fallback — so one family at two weight formats is two `.so` files, and branching on a quant
descriptor inside one plugin is the anti-pattern the split exists to prevent.

The in-tree plugins are the templates: `arch/llama_fp8/llama_fp8.cpp` (the smallest complete
decoder), `arch/gemma3`, `arch/qwen35_bf16`, and the header-only blocks in `arch/common/`
(`rad_block_attn*.h`, `rad_block_mlp*.h`, `rad_block_moe_fp8.h`, `rad_block_gdn*.h`,
`rad_block_dflash2.h`, `rad_block_mtp*.h`, `rad_block_vit.h`, `rad_block_ple.h`,
`rad_block_glm_*.h`, `rad_qsa.h`, …). Include those by `<arch/rad_arch.h>` — they are installed
beside the ABI and the in-tree plugins include them by that spelling.

## Exports

```c
const char* rad_arch_id(void);              /* "llama", "qwen35", ... */
const char* rad_arch_quant(void);           /* "" unquantised, else the descriptor served */
int         rad_arch_serves_imports(void);  /* optional: 0 declines an imported quantisation */
int         rad_arch_probe(const RadModelMeta* meta, RadArchProbe* out);   /* optional */
int         rad_arch_declare(RadBuilder* b, const RadModelMeta* meta, const RadBuildCtx* ctx);
void        rad_arch_step(struct RadCtx* c, const struct RadBatch* batch);
const RadChatFormat*  rad_arch_chat_format(const RadModelMeta* meta);      /* optional */
const RadAudioFormat* rad_arch_audio_format(const RadModelMeta* meta);     /* optional */
```

Everything but declare/step is optional. `RAD_ARCH_PLUGIN(NS, ID, QUANT, VERSION, DESC)` in
`arch/common/rad_arch.h` emits the whole export block, including `rad_plugin_info` (kind
`RAD_PLUGIN_ARCH`). The plugin's **name must be unique**; the architecture id may repeat.

`rad_arch_serves_imports` exists because a checkpoint whose quantisation was imported weight by
weight (Quark, ParoQuant, GPTQ, NVFP4) names its producing tool, which no plugin claims: the family's
one quantised plugin serves it, and two are refused by name. A second quantised plugin that reads
only its own containers returns 0.

## Declare: the metadata

`RadModelMeta` is what the loader read from the container's header before any plugin ran: `arch_id`,
`name`, `quant`, `n_layers`, `n_embd`, `n_head`, `n_head_kv`, **`head_dim`**, `n_ff`, `n_vocab`,
`n_ctx_train`, expert counts, `rms_eps`, `rope_theta`, and a free-form `kv_key`/`kv_val` table for
everything else (`rad_meta_geti/getf/gets`). Read dimensions from here — one plugin covers every size
in a family.

**`head_dim` is read, never derived.** On the Qwen hybrids it is 256 while `hidden / heads` is 213;
a derived value silently breaks projection widths, KV page size and the attention kernel's band.

`RadBuildCtx` is the deployment the plugin must declare against:

| field | meaning |
|---|---|
| `rank`, `world_size` | declare runs once per rank; **dimensions are already divided** — the core inserts no collectives |
| `max_tok`, `max_seqs`, `max_ctx`, `max_spec` | the step bounds: buffer sizing, kernel bands, the scheduler's window |
| `max_mixed_tok` | the most a multi-sequence step may carry; a wider step is one sequence (prefill span) |
| `tp_wire_lossy`, `tp_wire_min_bytes` | invert into `all_reduce`'s `exact`; below the floor a message must go exact |
| `kv_dtype`, `kv_quant` | the attention cache width and, under fp8, its narrow layout (`RAD_KVQ_V4`, `RAD_KVQ_K6V4`) |
| `attn_prefill_fp8` | `RAD_ATTN_FP8_*` bits to pass as `attn_paged`'s `prefill_fp8` |
| `kv_resident_blocks`, `kv_stream_ok` | KV streaming: the ring depth, and permission to declare the streamed-only ops |
| `state_dtype` | `RAD_F16` when `--state-cache-dtype f16` is served by this plugin |
| `is_draft`, `scope` | a drafter/encoder declare; `scope` prefixes declared names (`draft`, `vision`, `audio`) |
| `shape_probe` | a **sizing** declare at a smaller `max_tok`; must not touch state the real declare wrote |
| `max_enc_patches`, `max_audio_frames` | the encoder pass sizes |
| `max_out_rows` | logits rows the KL mode needs (0 = the plugin's own rule) |
| `cvec` | the loaded control vector, or null |
| `tp_attn_rows` | row-split prefill attention requested |
| `draft_coupling`, `draft_tree` | declare the coupled/tree forms |
| `weight_prefetch_off` | declare no weight prefetch beside collectives |

## Declare: weights

```c
static const RadWeightDecl d = {
    .dtype = RAD_BF16, .rank = 2, .shape = {n_embd, n_ff},
    .access = RAD_ACCESS_PER_TOKEN, .shard = RAD_SHARD_ROW,
    .group = grp_layer(l), .optional = 0,
    .n_row_parts = 0, .source = nullptr, .planes = nullptr,
};
rad_weight w = rad_decl_weight(b, "blk.0.mlp.gate.weight", &d);
```

- **`access`** tells the placement planner how the weight is read: `PER_TOKEN` (dense),
  `PER_REQUEST` (an encoder tower, read once per request), `CONDITIONAL` (routed experts),
  `RARE`, `VOCAB` (lm_head/embedding: read every token but huge, and one is a gather).
- **`shard`**: `NONE` (replicated), `ROW` (split dim 0), `COL` (split dim 1, needs a following
  all-reduce). The plugin says; the core does not infer. Uneven splits use
  `rad_weight_shard_span`/`_parts` (a width that does not divide into whole 128-column blocks; the
  delta net's `[q|k|v|z]` of unequal part widths).
- **`group`** is the unit of movement (`layer`, `expert`, `slot`); an expert's gate/up/down share one,
  because the unit of transfer is an expert, not a tensor.
- **`row_parts`** describes what a `SHARD_ROW` weight stacks along dim 0 (`[gate|up]`,
  `[q|k|v|z]`). **A tensor-parallel rank owns a slice of each stacked projection**, never a
  contiguous slice of the stack; the parts cannot be inferred from a fused container tensor, and a
  contiguous slice produces fluent wrong text.
- **`source`/`planes`** view an existing stored weight: a block-fp8 linear's `codes,scale` are one
  logical weight taken by a GEMM as two operands. Null `source` is the weight of its own name; null
  `planes` takes every plane.
- A dtype other than the plane's is legal only as an exact widening (a bf16 norm read as f32).

The **name map** (`rad_decl_name_map`) is the plugin's map from checkpoint tensor names to declared
weight names, and it is the only place convert-time fusions live: `COPY`, `CONCAT`, `WNORM` (a
weight-normalised pair folded to the plain weight), `ROWS`, `TRANSPOSE`, `EMA` (an EnCodec codebook),
`TAP`, `COLS`. `rad-convert` reads this map from the plugin, so it needs no knowledge of the model.
`RAD_MAP_COPY` with several sources means "first present wins" (a tied lm_head); `CONCAT` requires
all. Ask for a weight's encoding with `rad_weight_encoding` **after** its map entry when the graph
depends on the format (fp8 GEMM vs plain, rotated expert kernels).

## Declare: buffers and KV groups

`rad_decl_buffer` names every activation: `RAD_BUF_TRANSIENT` (liveness-analysed, may share bytes),
`RAD_BUF_PERSIST` (across steps: a calibration accumulator, a routing histogram), or
`RAD_BUF_DERIVED` (computed once per step by the core from the batch — `derive` names the
derivation, e.g. `conv_state_index`). Buffers live in one arena sized once, not an allocator per
step.

`rad_decl_kv_group` declares each state kind: `RAD_KV_FULL` (paged per-token, growing),
`RAD_KV_WINDOW` (bounded by a window), `RAD_KV_LINEAR` (per-sequence fixed state, GDN),
`RAD_KV_CONV` (a rolling window of `width-1+num_spec`), `RAD_KV_RING` (window floor set at runtime).
The **block size is not declared** — it comes from the resolved attention kernel's constraints; read
it back with `rad_kv_block_size`. `rad_bind_layer_kv` binds each layer so the block manager knows how
many state instances a sequence owns; `rad_kv_group_zero` (a rank with no heads),
`rad_kv_group_replicated` (every rank holds the same bytes), and `rad_kv_group_dense_read` (read over
the whole history every step, so it cannot stream) are the three refinements.

## Declare: ops

```c
rad_op h = RAD_OP(b, "gemm_w4a8_nt",
                  RAD_PARAMS(RAD_RANGE("M", 1, max_tok), RAD_INT("N", 8192), RAD_INT("K", 5120)),
                  RAD_WEIGHTS(w));
rad_op_reads (b, h, RAD_BUFS(b_x), n);
rad_op_writes(b, h, RAD_BUFS(b_y), n);
```

- Parameter names and operand order come from `docs/OPS.md`; a new name is allowed only when the
  vocabulary has none, and the first plugin to declare it fixes its schema.
- `RAD_RANGE` on exactly one parameter per op gives the selector its bucket table. The core carries
  every parameter opaquely except `RAD_PROLE_SEQ_CHUNK` (cut steps on it) and `RAD_PROLE_CAPACITY`.
- `rad_op_reads`/`rad_op_writes` are the buffer planner's liveness. An undeclared use is assumed live
  for the whole program: sound but wasteful.
- **Fusion is selection.** Ask for the fused op; if `rad_op_resolved` is false, emit the chain. A
  fusion's `replaces` hook is a checkable equivalence, not a rewrite.
- `rad_op_declared(b, "op")` asks whether any library declares an optional op; use it before
  emitting a fused form only an optional library defines.
- Buffers touched from the **second lane** go through `rad_buf_concurrent`, and an architecture whose
  lane-1 work lies in one stretch declares `rad_lane1_window_begin/end` around it. The runtime
  refuses a lane-1 op outside the window; the window lets buffers used only outside it share bytes.

Step output: `rad_declare_logits` names the buffer the sampler reads (its dim 0 bounds `n_out`; a
[n_tok, vocab] logits buffer at a large vocabulary is gigabytes — declare the gather +
`logits_gemm` shape instead). An embedding program declares `rad_declare_pooled` instead (one row a
sequence, no sampler, no generation). `rad_declare_scores` opts into host-side scored steps.

## Declare: drafters, encoders, generators

- `rad_declare_drafter(RadDrafterDecl)` — depth, block, proposal shape. `rad_drafter_shared_kv` (a
  drafter with no cache of its own), `rad_drafter_tree` (a tree over the proposals),
  `rad_drafter_short_block` (a second, shorter pass) are refinements. Declare the drafter **last**.
- `rad_declare_encoder(RadEncoderDecl)` — a vision/audio tower, media preprocessed by the core.
- `rad_declare_generator` / `rad_declare_audio_generator` / `rad_declare_video_generator` — image,
  audio and video generation programs; the engine sizes the step from the probe's `gen_*` fields
  before declare.
- Optional format exports let a model state what it knows and the server learn nothing new:
  `rad_arch_chat_format` (a data description of reasoning/tool-call markup, replacing template
  analysis), `rad_arch_audio_format` (how a speech model hears: clip tokens, prompt, language,
  transcript start), `rad_arch_text_prepare` (rewrite request text into training form),
  `rad_arch_video_positions`/`rad_arch_video_pack` (video pass geometry and packing),
  `rad_arch_encode_geom` (encoder pass geometry). Each struct grows only at its end behind
  `struct_size`, so adding a field does not move the ABI.

**Media preprocessing is the core's**, not the plugin's: resizing, patching, placeholder expansion
and positions in six processor families (Qwen-VL, Gemma 4, LFM2-VL, Unlimited-OCR, GLM-5.3, Pixtral
for Nemotron), chosen by `vision_config.model_type` plus owning-model checks; audio through Gemma 4's
extractor or Qwen3-ASR's Whisper one, chosen by `audio_config.model_type`. A vision model with another
pipeline needs a core change.

## Step: issuing the graph

`RadBatch` is built once per step and shared by every block: `phase` (`PREFILL`/`DECODE`/`MIXED`),
`n_tok`, `n_seq`, `token_ids`, `positions`, `cu_seqlens`, `seq_ids`, `lora_slot`, `n_out`/`out_ids`,
`q_lens`/`ctx_lens`/`max_q_len`/`max_ctx_len`, the KV-group array, the speculative fields
(`n_spec`, `draft_pass`, …).

```c
static void step(RadCtx* c, const RadBatch* batch) {
    if (batch->draft_pass > 0) { m.ds.step(c, batch); return; }      /* query pass */
    if (batch->draft_pass < 0) { m.ds.context(c, batch); return; }   /* context pass */
    RAD_ISSUE(c, m.op_embed, praw(batch->token_ids, RAD_I32, batch->n_tok),
              RAD_W(m.w_tok), brows(m.b_x, batch->n_tok));
    for (auto& layer : m.layers) { layer.attn.step(c, batch); layer.mlp.step(c, batch); }
    /* ...heads; the lm_head only over batch->n_out rows... */
}
```

- `rad_issue(c, op, operands…, n)` / `RAD_ISSUE` (n = the batch's ranged extent) /
  `RAD_ISSUE_N` (an explicit row count). `rad_step_fail(c, what)` sets the status helpers return.
- **A mixed step is the trap.** Continuous batching produces steps with a prefill chunk beside decode
  rows. A block that issues one path for the whole step sends its decode rows down the chunked path,
  which commits recurrent state directly and takes no `num_accepted`; a rejected draft cannot be
  rolled back, and the model stays fluent while drifting. Issue the recurrent path over
  `RadBatch::n_seq_decode` sequences and the chunked path over the rest (the core orders steps
  decode-first, so each half is contiguous; `cu_seqlens` holds absolute offsets, so pass
  `cu + n_seq_decode` with the token operands at their base).
- A step issued on lane 1 outside the declared window fails by name. An op the core appends behind a
  declared one (runtime LoRA) counts at that op's position.
- **Zero rows is not a call**: issuing an op over zero rows (a zero ranged extent, or all-empty
  sequences) returns `RAD_E_SHAPE` and writes nothing. The caller skips the issue, as the
  architectures' decode and prefill halves do.
- `n_out` is 0 for a chunk that finishes no sequence — do not run the lm_head.

## Probe: what the plugin measured

`rad_arch_probe(meta, out)` runs before declare. Its fields are the engine's permissions:

| field | effect |
|---|---|
| `draft_depth`, `draft_depth_fixed` | the container's spec depth; a block drafter's trained block is the only depth it serves |
| `shape_probe_ok` | the plugin tolerates a sizing declare, so the activation arena can be lent |
| `enc_merge`, `audio_frames` | encoder pass granularity; longest audio clip |
| `gen_rows`, `gen_patches`, `gen_audio`, `gen_ctx`, `gen_video` | generator sizing before declare |
| `kv_quant_ok` | bits of the narrow KV layouts this model's kernels serve |
| `chain_pays`, `tree_pays` | the plugin measured chained launches / draft trees faster |
| `kv_stream_ok` | the architecture can stream a KV group (sparse, block-addressed, with `kv_resolve`) |
| `control_vector_ok` | the plugin applies `RadBuildCtx::cvec` |
| `tp_attn_rows_ok` | the plugin serves row-split prefill attention |
| `state_f16_ok` | the plugin declares recurrent state at f16 and its kernels read it |

A flag whose probe bit is 0 is **refused at startup with the plugin named** (`--kv-cache-dtype
fp8_v4`, `--control-vector`, `--tp-attn-split rows`, `--state-cache-dtype f16`, `--kv-resident`).
That is the design: a deployment switch the plugin cannot honour is an error, never a silent
substitution.

## Worked pattern: the smallest complete plugin

`arch/llama_fp8/llama_fp8.cpp` builds twice from one source — `llama_fp8` (quant `fp8_e4m3`) and
`llama_dense_fp8` (quant `""`), differing only in the descriptor and the graph-dump text. It shows
the whole shape:

1. `struct Layer { AttnFP8 attn; MlpFP8 mlp; }`, `struct Model` holding `Names`, `Geom`, buffer and
   op handles.
2. `declare()`: name map entries first, then weights (with `rad_weight_encoding` where the head's
   format decides the kernel), buffers, KV group, the embedding and its all-reduce, then each layer's
   `declare()` from the shared blocks, the final norm, gather and lm_head, **the drafter last**, and
   `rad_note` lines explaining the shape.
3. `step()`: draft passes first (sign selects query/context), then embedding (+ exact all-reduce
   when needed), each layer's `step()`, the final norm, and the head only over `n_out`.
4. `probe()`: reads `draft.block_size` from the metadata and refuses any other depth for a block
   drafter.
5. `RAD_ARCH_PLUGIN(...)` at the bottom.

Add a new family by copying a neighbour, replacing the name map and geometry, and composing the
blocks you need; add a new *format* by cloning the plugin with another `(id, quant)` pair and the
ops that format needs.

## Verifying a new plugin

```sh
radiance --model m.rad --debug-graph          # every op, its resolved kernel, and why
radiance --model m.rad --debug-placement      # the plan
rad-convert $SRC -o m.rad --plan-only -v      # a convert that plans will boot
rad-kbench -m m.rad --kernels mylib,libr4d,libref --report k.md --bench
rad-tune -m m.rad --tp 2                      # only at the shapes you serve
```

`tests/arch_test.cpp` drives a plugin's builder the way the core does, including lane-window
liveness, so the pinned graphs and steps are the ones the core plans. A plugin that declares a weight
form no kernel `layout` accepts is a declare-time error naming the weight, encoding and kernel — the
same check `rad-convert` runs before writing.

## Related

- [[Radiance Plugins]] — kernel libraries, encodings, the build path, ABI law.
- [[Radiance Conversion]] — how the name map drives a convert, and `--set`/descriptor selection.
- [[Radiance Model Families]] — the existing `(id, quant)` pairs and their recipes.
- [[Radiance Tools]] — `rad-kbench`, `rad-tune`, `rad-info --plugins`.