serverAppConfigs/serverAppConfigs-docs/LLM/Radiance Architecture Plugins.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
howto	Radiance Architecture Plugins — declaring a model	Writing a radiance architecture plugin: declaring weights, buffers, KV groups and ops, then issuing the step.	llm radiance plugins howto	2026-10-30

Radiance architecture plugins

An architecture plugin declares a model in two phases: rad_arch_declare names every weight, buffer, KV group and op once, against the model's metadata; rad_arch_step issues the ops for one step. The engine picks the plugin by the exact (rad_arch_id, rad_arch_quant) pair — no fallback — so one family at two weight formats is two .so files, and branching on a quant descriptor inside one plugin is the anti-pattern the split exists to prevent.

The in-tree plugins are the templates: arch/llama_fp8/llama_fp8.cpp (the smallest complete decoder), arch/gemma3, arch/qwen35_bf16, and the header-only blocks in arch/common/ (rad_block_attn*.h, rad_block_mlp*.h, rad_block_moe_fp8.h, rad_block_gdn*.h, rad_block_dflash2.h, rad_block_mtp*.h, rad_block_vit.h, rad_block_ple.h, rad_block_glm_*.h, rad_qsa.h, …). Include those by <arch/rad_arch.h> — they are installed beside the ABI and the in-tree plugins include them by that spelling.

Exports

const char* rad_arch_id(void);              /* "llama", "qwen35", ... */
const char* rad_arch_quant(void);           /* "" unquantised, else the descriptor served */
int         rad_arch_serves_imports(void);  /* optional: 0 declines an imported quantisation */
int         rad_arch_probe(const RadModelMeta* meta, RadArchProbe* out);   /* optional */
int         rad_arch_declare(RadBuilder* b, const RadModelMeta* meta, const RadBuildCtx* ctx);
void        rad_arch_step(struct RadCtx* c, const struct RadBatch* batch);
const RadChatFormat*  rad_arch_chat_format(const RadModelMeta* meta);      /* optional */
const RadAudioFormat* rad_arch_audio_format(const RadModelMeta* meta);     /* optional */

Everything but declare/step is optional. RAD_ARCH_PLUGIN(NS, ID, QUANT, VERSION, DESC) in arch/common/rad_arch.h emits the whole export block, including rad_plugin_info (kind RAD_PLUGIN_ARCH). The plugin's name must be unique; the architecture id may repeat.

rad_arch_serves_imports exists because a checkpoint whose quantisation was imported weight by weight (Quark, ParoQuant, GPTQ, NVFP4) names its producing tool, which no plugin claims: the family's one quantised plugin serves it, and two are refused by name. A second quantised plugin that reads only its own containers returns 0.

Declare: the metadata

RadModelMeta is what the loader read from the container's header before any plugin ran: arch_id, name, quant, n_layers, n_embd, n_head, n_head_kv, head_dim, n_ff, n_vocab, n_ctx_train, expert counts, rms_eps, rope_theta, and a free-form kv_key/kv_val table for everything else (rad_meta_geti/getf/gets). Read dimensions from here — one plugin covers every size in a family.

head_dim is read, never derived. On the Qwen hybrids it is 256 while hidden / heads is 213; a derived value silently breaks projection widths, KV page size and the attention kernel's band.

RadBuildCtx is the deployment the plugin must declare against:

field	meaning
rank, world_size	declare runs once per rank; dimensions are already divided — the core inserts no collectives
max_tok, max_seqs, max_ctx, max_spec	the step bounds: buffer sizing, kernel bands, the scheduler's window
max_mixed_tok	the most a multi-sequence step may carry; a wider step is one sequence (prefill span)
tp_wire_lossy, tp_wire_min_bytes	invert into all_reduce's exact; below the floor a message must go exact
kv_dtype, kv_quant	the attention cache width and, under fp8, its narrow layout (RAD_KVQ_V4, RAD_KVQ_K6V4)
attn_prefill_fp8	RAD_ATTN_FP8_* bits to pass as attn_paged's prefill_fp8
kv_resident_blocks, kv_stream_ok	KV streaming: the ring depth, and permission to declare the streamed-only ops
state_dtype	RAD_F16 when --state-cache-dtype f16 is served by this plugin
is_draft, scope	a drafter/encoder declare; scope prefixes declared names (draft, vision, audio)
shape_probe	a sizing declare at a smaller max_tok; must not touch state the real declare wrote
max_enc_patches, max_audio_frames	the encoder pass sizes
max_out_rows	logits rows the KL mode needs (0 = the plugin's own rule)
cvec	the loaded control vector, or null
tp_attn_rows	row-split prefill attention requested
draft_coupling, draft_tree	declare the coupled/tree forms
weight_prefetch_off	declare no weight prefetch beside collectives

Declare: weights

static const RadWeightDecl d = {
    .dtype = RAD_BF16, .rank = 2, .shape = {n_embd, n_ff},
    .access = RAD_ACCESS_PER_TOKEN, .shard = RAD_SHARD_ROW,
    .group = grp_layer(l), .optional = 0,
    .n_row_parts = 0, .source = nullptr, .planes = nullptr,
};
rad_weight w = rad_decl_weight(b, "blk.0.mlp.gate.weight", &d);

• access tells the placement planner how the weight is read: PER_TOKEN (dense), PER_REQUEST (an encoder tower, read once per request), CONDITIONAL (routed experts), RARE, VOCAB (lm_head/embedding: read every token but huge, and one is a gather).
• shard: NONE (replicated), ROW (split dim 0), COL (split dim 1, needs a following all-reduce). The plugin says; the core does not infer. Uneven splits use rad_weight_shard_span/_parts (a width that does not divide into whole 128-column blocks; the delta net's [q|k|v|z] of unequal part widths).
• group is the unit of movement (layer, expert, slot); an expert's gate/up/down share one, because the unit of transfer is an expert, not a tensor.
• row_parts describes what a SHARD_ROW weight stacks along dim 0 ([gate|up], [q|k|v|z]). A tensor-parallel rank owns a slice of each stacked projection, never a contiguous slice of the stack; the parts cannot be inferred from a fused container tensor, and a contiguous slice produces fluent wrong text.
• source/planes view an existing stored weight: a block-fp8 linear's codes,scale are one logical weight taken by a GEMM as two operands. Null source is the weight of its own name; null planes takes every plane.
• A dtype other than the plane's is legal only as an exact widening (a bf16 norm read as f32).

The name map (rad_decl_name_map) is the plugin's map from checkpoint tensor names to declared weight names, and it is the only place convert-time fusions live: COPY, CONCAT, WNORM (a weight-normalised pair folded to the plain weight), ROWS, TRANSPOSE, EMA (an EnCodec codebook), TAP, COLS. rad-convert reads this map from the plugin, so it needs no knowledge of the model. RAD_MAP_COPY with several sources means "first present wins" (a tied lm_head); CONCAT requires all. Ask for a weight's encoding with rad_weight_encoding after its map entry when the graph depends on the format (fp8 GEMM vs plain, rotated expert kernels).

Declare: buffers and KV groups

rad_decl_buffer names every activation: RAD_BUF_TRANSIENT (liveness-analysed, may share bytes), RAD_BUF_PERSIST (across steps: a calibration accumulator, a routing histogram), or RAD_BUF_DERIVED (computed once per step by the core from the batch — derive names the derivation, e.g. conv_state_index). Buffers live in one arena sized once, not an allocator per step.

rad_decl_kv_group declares each state kind: RAD_KV_FULL (paged per-token, growing), RAD_KV_WINDOW (bounded by a window), RAD_KV_LINEAR (per-sequence fixed state, GDN), RAD_KV_CONV (a rolling window of width-1+num_spec), RAD_KV_RING (window floor set at runtime). The block size is not declared — it comes from the resolved attention kernel's constraints; read it back with rad_kv_block_size. rad_bind_layer_kv binds each layer so the block manager knows how many state instances a sequence owns; rad_kv_group_zero (a rank with no heads), rad_kv_group_replicated (every rank holds the same bytes), and rad_kv_group_dense_read (read over the whole history every step, so it cannot stream) are the three refinements.

Declare: ops

rad_op h = RAD_OP(b, "gemm_w4a8_nt",
                  RAD_PARAMS(RAD_RANGE("M", 1, max_tok), RAD_INT("N", 8192), RAD_INT("K", 5120)),
                  RAD_WEIGHTS(w));
rad_op_reads (b, h, RAD_BUFS(b_x), n);
rad_op_writes(b, h, RAD_BUFS(b_y), n);

• Parameter names and operand order come from docs/OPS.md; a new name is allowed only when the vocabulary has none, and the first plugin to declare it fixes its schema.
• RAD_RANGE on exactly one parameter per op gives the selector its bucket table. The core carries every parameter opaquely except RAD_PROLE_SEQ_CHUNK (cut steps on it) and RAD_PROLE_CAPACITY.
• rad_op_reads/rad_op_writes are the buffer planner's liveness. An undeclared use is assumed live for the whole program: sound but wasteful.
• Fusion is selection. Ask for the fused op; if rad_op_resolved is false, emit the chain. A fusion's replaces hook is a checkable equivalence, not a rewrite.
• rad_op_declared(b, "op") asks whether any library declares an optional op; use it before emitting a fused form only an optional library defines.
• Buffers touched from the second lane go through rad_buf_concurrent, and an architecture whose lane-1 work lies in one stretch declares rad_lane1_window_begin/end around it. The runtime refuses a lane-1 op outside the window; the window lets buffers used only outside it share bytes.

Step output: rad_declare_logits names the buffer the sampler reads (its dim 0 bounds n_out; a [n_tok, vocab] logits buffer at a large vocabulary is gigabytes — declare the gather + logits_gemm shape instead). An embedding program declares rad_declare_pooled instead (one row a sequence, no sampler, no generation). rad_declare_scores opts into host-side scored steps.

Declare: drafters, encoders, generators

• rad_declare_drafter(RadDrafterDecl) — depth, block, proposal shape. rad_drafter_shared_kv (a drafter with no cache of its own), rad_drafter_tree (a tree over the proposals), rad_drafter_short_block (a second, shorter pass) are refinements. Declare the drafter last.
• rad_declare_encoder(RadEncoderDecl) — a vision/audio tower, media preprocessed by the core.
• rad_declare_generator / rad_declare_audio_generator / rad_declare_video_generator — image, audio and video generation programs; the engine sizes the step from the probe's gen_* fields before declare.
• Optional format exports let a model state what it knows and the server learn nothing new: rad_arch_chat_format (a data description of reasoning/tool-call markup, replacing template analysis), rad_arch_audio_format (how a speech model hears: clip tokens, prompt, language, transcript start), rad_arch_text_prepare (rewrite request text into training form), rad_arch_video_positions/rad_arch_video_pack (video pass geometry and packing), rad_arch_encode_geom (encoder pass geometry). Each struct grows only at its end behind struct_size, so adding a field does not move the ABI.

Media preprocessing is the core's, not the plugin's: resizing, patching, placeholder expansion and positions in six processor families (Qwen-VL, Gemma 4, LFM2-VL, Unlimited-OCR, GLM-5.3, Pixtral for Nemotron), chosen by vision_config.model_type plus owning-model checks; audio through Gemma 4's extractor or Qwen3-ASR's Whisper one, chosen by audio_config.model_type. A vision model with another pipeline needs a core change.

Step: issuing the graph

RadBatch is built once per step and shared by every block: phase (PREFILL/DECODE/MIXED), n_tok, n_seq, token_ids, positions, cu_seqlens, seq_ids, lora_slot, n_out/out_ids, q_lens/ctx_lens/max_q_len/max_ctx_len, the KV-group array, the speculative fields (n_spec, draft_pass, …).

static void step(RadCtx* c, const RadBatch* batch) {
    if (batch->draft_pass > 0) { m.ds.step(c, batch); return; }      /* query pass */
    if (batch->draft_pass < 0) { m.ds.context(c, batch); return; }   /* context pass */
    RAD_ISSUE(c, m.op_embed, praw(batch->token_ids, RAD_I32, batch->n_tok),
              RAD_W(m.w_tok), brows(m.b_x, batch->n_tok));
    for (auto& layer : m.layers) { layer.attn.step(c, batch); layer.mlp.step(c, batch); }
    /* ...heads; the lm_head only over batch->n_out rows... */
}

• rad_issue(c, op, operands…, n) / RAD_ISSUE (n = the batch's ranged extent) / RAD_ISSUE_N (an explicit row count). rad_step_fail(c, what) sets the status helpers return.
• A mixed step is the trap. Continuous batching produces steps with a prefill chunk beside decode rows. A block that issues one path for the whole step sends its decode rows down the chunked path, which commits recurrent state directly and takes no num_accepted; a rejected draft cannot be rolled back, and the model stays fluent while drifting. Issue the recurrent path over RadBatch::n_seq_decode sequences and the chunked path over the rest (the core orders steps decode-first, so each half is contiguous; cu_seqlens holds absolute offsets, so pass cu + n_seq_decode with the token operands at their base).
• A step issued on lane 1 outside the declared window fails by name. An op the core appends behind a declared one (runtime LoRA) counts at that op's position.
• Zero rows is not a call: issuing an op over zero rows (a zero ranged extent, or all-empty sequences) returns RAD_E_SHAPE and writes nothing. The caller skips the issue, as the architectures' decode and prefill halves do.
• n_out is 0 for a chunk that finishes no sequence — do not run the lm_head.

Probe: what the plugin measured

rad_arch_probe(meta, out) runs before declare. Its fields are the engine's permissions:

field	effect
draft_depth, draft_depth_fixed	the container's spec depth; a block drafter's trained block is the only depth it serves
shape_probe_ok	the plugin tolerates a sizing declare, so the activation arena can be lent
enc_merge, audio_frames	encoder pass granularity; longest audio clip
gen_rows, gen_patches, gen_audio, gen_ctx, gen_video	generator sizing before declare
kv_quant_ok	bits of the narrow KV layouts this model's kernels serve
chain_pays, tree_pays	the plugin measured chained launches / draft trees faster
kv_stream_ok	the architecture can stream a KV group (sparse, block-addressed, with kv_resolve)
control_vector_ok	the plugin applies RadBuildCtx::cvec
tp_attn_rows_ok	the plugin serves row-split prefill attention
state_f16_ok	the plugin declares recurrent state at f16 and its kernels read it

A flag whose probe bit is 0 is refused at startup with the plugin named (--kv-cache-dtype fp8_v4, --control-vector, --tp-attn-split rows, --state-cache-dtype f16, --kv-resident). That is the design: a deployment switch the plugin cannot honour is an error, never a silent substitution.

Worked pattern: the smallest complete plugin

arch/llama_fp8/llama_fp8.cpp builds twice from one source — llama_fp8 (quant fp8_e4m3) and llama_dense_fp8 (quant ""), differing only in the descriptor and the graph-dump text. It shows the whole shape:

1. struct Layer { AttnFP8 attn; MlpFP8 mlp; }, struct Model holding Names, Geom, buffer and op handles.
2. declare(): name map entries first, then weights (with rad_weight_encoding where the head's format decides the kernel), buffers, KV group, the embedding and its all-reduce, then each layer's declare() from the shared blocks, the final norm, gather and lm_head, the drafter last, and rad_note lines explaining the shape.
3. step(): draft passes first (sign selects query/context), then embedding (+ exact all-reduce when needed), each layer's step(), the final norm, and the head only over n_out.
4. probe(): reads draft.block_size from the metadata and refuses any other depth for a block drafter.
5. RAD_ARCH_PLUGIN(...) at the bottom.

Add a new family by copying a neighbour, replacing the name map and geometry, and composing the blocks you need; add a new format by cloning the plugin with another (id, quant) pair and the ops that format needs.

Verifying a new plugin

radiance --model m.rad --debug-graph          # every op, its resolved kernel, and why
radiance --model m.rad --debug-placement      # the plan
rad-convert $SRC -o m.rad --plan-only -v      # a convert that plans will boot
rad-kbench -m m.rad --kernels mylib,libr4d,libref --report k.md --bench
rad-tune -m m.rad --tp 2                      # only at the shapes you serve

tests/arch_test.cpp drives a plugin's builder the way the core does, including lane-window liveness, so the pinned graphs and steps are the ones the core plans. A plugin that declares a weight form no kernel layout accepts is a declare-time error naming the weight, encoding and kernel — the same check rad-convert runs before writing.

Related

• Radiance Plugins — kernel libraries, encodings, the build path, ABI law.
• Radiance Conversion — how the name map drives a convert, and --set/descriptor selection.
• Radiance Model Families — the existing (id, quant) pairs and their recipes.
• Radiance Tools — rad-kbench, rad-tune, rad-info --plugins.