serverAppConfigs/serverAppConfigs-docs/LLM/Radiance Plugins.md

---
type: "howto"
title: "Radiance Plugins — kernel libraries and quantisers"
description: "Writing radiance kernel and quantiser plugins: the ABI, kernel rows, hooks, encodings, and the build path."
tags:
  - llm
  - radiance
  - plugins
  - howto
timestamp: 2026-10-30
---

# Radiance plugins — kernels and quantisers

Three kinds of `.so` extend the engine, each exporting a few C symbols and linking **nothing of
radiance's** (they compile against the installed `abi/` headers):

| kind | value | contract |
|---|---|---|
| kernel library | `RAD_PLUGIN_KERNEL` (1) | ops for some hardware: a card family, the CPU, an oracle |
| architecture | `RAD_PLUGIN_ARCH` (2) | one model family at one weight format — see [[Radiance Architecture Plugins]] |
| quantiser | `RAD_PLUGIN_QUANT` (3) | weights → encoding planes, for `rad-convert` |

In-tree plugins use the same interface a third party does: each builds with `rad_add_plugin` and is
found and ranked by what it declares, never by its name. `ctest -L oot` rebuilds every bundled
plugin out of tree against an install.

## Identity and ABI

```c
#include "rad_abi.h"   /* required; pulls rad_types.h and rad_encoding.h */

uint32_t rad_plugin_abi_version(void) { return RAD_ABI_VERSION; }

const RadPluginInfo* rad_plugin_info(void) {
    static const RadPluginInfo i = {
        RAD_PLUGIN_KERNEL, "mykernels", "0.1", "a GEMM or two", "gfx1100",
    };
    return &i;
}
```

- `RAD_ABI_VERSION` (31 at this commit) is bumped on any shape change in a header a plugin compiles
  against. Return the macro; a mismatch is refused at load with both numbers printed.
- `build_target` is printed, not trusted: the targets the library was built for (`"gfx1100"`,
  `"gfx11-generic"`, several separated by spaces/commas), `"host"`, or `""` for portable. The engine
  checks the library's actual code objects instead (§"A library for a new card").
- Optional exports, resolved by exact name: `rad_plugin_open` (return `RAD_E_UNSUPPORTED` to decline
  this machine and unload), `rad_plugin_close`, `rad_kernel_concurrent`, `rad_kernel_collective`,
  `rad_plugin_reference`, `rad_kernel_reference`, `rad_kernel_chain`.

## What a kernel library exports

```c
int                rad_kernel_schema_count(void);
const RadOpSchema* rad_kernel_schema_at(int i);
int                rad_kernel_count(void);
const RadKernelInfo* rad_kernel_at(int i);
```

A row declares one op implementation and what it can serve:

```c
static const RadConstraint c_gemm[] = {
    RAD_CLE("M", 64),              /* M <= 64            */
    RAD_CDIV("K", 128),            /* K divisible by 128 */
    RAD_CIN("dtype", "fp8a8 w4a8") /* dtype in that set  */
};

static const RadKernelInfo k[] = {{
    .name = "mygemm_m64", .op = "gemm_nt_q", .family = "gemm",
    .computes = "row-major A times column-major B, fp8 in, bf16 out",
    .shape    = "M<=64, K%128==0",
    .dtypes   = "fp8a8, w4a8",
    .domain   = RAD_DOMAIN_DEVICE,
    .priority = 50,
    .constraints = c_gemm, .n_constraints = 3,
    .launch = my_launch,
}};
```

- Operators: `RAD_CEQ`, `RAD_CLE`, `RAD_CGE`, `RAD_CDIV`, `RAD_CIN`. The engine takes bucket
  boundaries from `LE`/`GE`/`EQ` and resolves once per band; `DIV` and `IN` are not intervals.
- **Declare only the band you serve.** The engine believes the claim; a kernel that says `M <= 2048`
  but is correct to 64 is wrong at concurrency.
- `device.*` keys are matched against what the engine measured about the ranks' cards, not against
  op parameters: `RAD_CEQ(RAD_FACT_P2P, 1)` means the kernel touches another card's memory. libr4d
  pairs every collective with a `_host` twin requiring `device.p2p == 0`, so exactly one matches any
  machine.
- **Priority breaks ties within one plugin**, highest first, then declaration order. **Between
  plugins the hierarchy is absolute**: the first library in `--kernels` order with any matching row
  supplies the op, however specialised a later library's row is. That makes a user's library an
  override, not a competitor.
- A derivative parameter (`RAD_DERIVED`) is supplied by the kernel; a constraint on a key the
  geometry omits does not hold during matching, and the winner's `RAD_C_EQ` writes it into the
  resolved geometry. `RAD_PROLE_SEQ_CHUNK` marks the parameter the scheduler must cut steps on;
  `RAD_PROLE_CAPACITY` marks a count the kernel only bounds itself by.

## Operands and extents

`RadArgs.t` is positional in the schema's order; each `RadTensor` carries the extent the caller
narrowed it to, usually smaller than the buffer. Size loops and grids from those extents, never from
a declared maximum. Where an op takes `cu_seqlens`, derive the sequence count from its own length
(`n = len - 1`) and each sequence's rows from `cu[n]` and `cu[n+1]`.

An absent optional operand is a **null `RadTensor.data`**, not a shorter operand count.
`rad_arg_in(a, i)` fetches one correctly; `rad_tensor_rows`, `rad_tensor_lastdim`,
`rad_tensor_row_pitch`, `rad_args_geti_or`, `rad_args_getf_or`, `rad_args_gets_or` are the other
everyday helpers in `rad_plugin.h`.

## Optional hooks

| hook | meaning |
|---|---|
| `init` / `fini` | per-resolved-instance state; `RadArgs.instance` hands it back. `init` runs once per (op, band, domain) after selection and before load |
| `scratch` | bytes of scratch one launch needs; sized at declare from the band's upper bound |
| `layout` / `relayout` | the kernel stores this weight in its own order, and how the loader produces it from the canonical planes |
| `unrelayout` | the inverse, so a tool holding stored bytes can compare against a dense oracle |
| `opd_shape` | size and fill every operand, so a tool can call the kernel cold |
| `replaces` | the chain of conventional ops this kernel computes in one launch |
| `tunables` / `tune_valid` | the instantiation axes, and which combinations are legal at a shape |

`replaces` must be **exact** and is compared byte for byte against the chain — a kernel that is only
close publishes nothing. `tunables` declares axes and legal values, not named variants; the core
takes the cross product, and `deflt` must be launchable (the loader refuses the plugin otherwise).
`rad-tune` measures survivors and writes the cache the selector reads; the chosen point arrives as
ordinary parameters (`rad_args_geti("sk")`), with no variant index.

`layout` answers: `RAD_OK` (own form, `relayout` will produce it), `RAD_E_UNSUPPORTED` (stored as
is — the common case, and the only form that can be read in place from the file/mapped tier),
`RAD_E_DTYPE` (this encoding is not mine — a declare-time error), `RAD_E_SHAPE`, or `RAD_LAYOUT_ANY`
(reads the bytes without interpreting them; claims no storage). A weight only `LAYOUT_ANY` readers
touch is stored as it is.

## What a launch may do

On a card the `RadStream` is **the engine's own AQL queue**, not a `hipStream_t`. The engine binary
defines and serves these HIP entry points:

- `hipLaunchKernel` (and `<<<>>>`, `hipLaunchKernelGGL`);
- `hipMemcpyAsync`, `hipMemcpy2DAsync`, `hipMemsetAsync`;
- `hipStreamSynchronize` (not from a launch — a launch must not synchronise);
- `hipMemset` (runs on the engine's queue and waits);
- `hipGetLastError` / `hipPeekAtLastError`, with HIP's semantics.

**Nothing else in HIP takes an engine stream.** `hipModuleLaunchKernel`, `hipExtLaunchKernel`,
cooperative launches, `hipLaunchHostFunc`, and libraries that launch through them (rocBLAS,
hipBLASLt, MIOpen, hipRTC) would hand HIP a queue it does not know. Header-only device code
(Composable Kernel templates, rocPRIM primitives) compiled into your `.hip` works. `hipEventRecord`
and `hipStreamWaitEvent` on an engine stream are refused by name.

The engine also reads the code object, which limits a kernel to: **no hostcall** (`printf`/`assert`
are refused at the first launch), **no dynamic stack**, **≤ 768 bytes of kernel arguments**, **≤ 64
KiB static+dynamic LDS per dispatch**, and an **uncompressed fat binary**. Device memory a kernel
needs comes from the `scratch` hook, never `hipMalloc` — the engine's allocator knows what is left.

`rad_plugin.h`'s conversions define the result: the reference contract is f32 accumulation, IEEE
round-to-nearest-even, **no FMA and no reassociation**. A plugin whose bf16 narrowing truncates
cannot be checked against the reference. `rad_load_f32`/`rad_store_f32`/`rad_load_bits`/
`rad_store_bits` read and write any core dtype from a plane.

## Colours of agreement

- `rad_kernel_concurrent(i)` — kernel `i` touches only its operands and scratch, so independent
  launches may overlap.
- `rad_kernel_collective(i)` — it exchanges data with other ranks; the engine keeps bulk link
  traffic away. `2` promises more: the launch ends the hold itself once the other ranks' bytes have
  landed, through a `link_gate` word it reaches at system scope; returning `RAD_LINK_ENDED` says it
  did.
- `rad_kernel_chain(i)` — the kernel can wait in its own prologue instead of behind the packet
  processor's barrier bit. Chained forms are recognised by `r4dchain` in the name and use a
  `RAD_CHAIN_REC_BYTES` record at `RAD_CHAIN_KARG_OFF`.
- `rad_plugin_reference` / `rad_kernel_reference` — an oracle library (or an oracle row beside a
  device row). A reference library ranks below every unnamed library, is refused for serving unless
  `--debug-accept-reference-kernels`, and is what `rad-kbench` checks against.

## Defining a new op

Only when `docs/OPS.md` has no name for it. The vocabulary document owns the rules: what a schema
must declare, and that **the first plugin in hierarchy order to declare an op fixes its schema** — a
later plugin declaring it differently is refused at load, both named. `ctest -R ops_doc` checks the
document against the registries, so the rules live there only. An op in `docs/OPS.md` is describable
by `libref`, and therefore checkable, even when your kernel cannot describe itself; an op you invent
runs only on a library that implements it.

A schema is positional data, not code:

```c
static const RadParamSpec params[] = {
    {"M", RAD_P_INT, RAD_REQUIRED, RAD_PROLE_NONE},   /* the ranged extent */
    {"n", RAD_P_INT, RAD_REQUIRED, RAD_PROLE_NONE},
    {"eps", RAD_P_F64, RAD_REQUIRED, RAD_PROLE_NONE},
    {"dtype", RAD_P_STR, RAD_REQUIRED, RAD_PROLE_NONE},
};
static const RadOperandSpec opds[] = {
    {"x", RAD_OPD_IN, 0}, {"w", RAD_OPD_WEIGHT, 1}, {"y", RAD_OPD_OUT, 0},
};
static const RadOpSchema schemas[] = {
    {"my_op", params, 4, opds, 3, "what it computes, for a human"},
};
```

`rad_kernel_schema_count/at` return this table. `RAD_REQUIRED`/`RAD_OPTIONAL`/`RAD_DERIVED` and the
operand roles (`IN`, `OUT`, `INOUT`, `WEIGHT`, `WTAB`) are the whole vocabulary; the kernel's
constraints and launch do the rest.

## Quantiser plugins

`abi/rad_quant.h` is the whole contract:

```c
int                     rad_quant_count(void);
const RadQuantizerInfo* rad_quant_at(int i);

typedef struct RadQuantizerInfo {
    const char*           name;       /* what a recipe names */
    const char*           doc;
    const RadQuantOption* options;    /* key, RAD_P_INT|F64|STR, default text, doc */
    int                   n_options;
    RadQuantEncodingFn    encoding;   /* the encoding this quantiser would write */
    RadQuantRowBlockFn    row_block;  /* rows per quantize() call; 0 = whole weight */
    RadQuantizeFn         quantize;   /* logical rows -> planes */
    RadQuantDecodeFn      decode;     /* optional, only for a scheme of its own */
} RadQuantizerInfo;
```

The flow for each weight a recipe rule picks: ask `encoding` (given name, shape, and — at declare —
no movement group), let the architecture declare against that encoding, then hand the weight over in
row blocks, so a 51 GB table streams. Other calibration data is the quantiser's own option
(`calib=DIR`); the output is **canonical planes only**, so every kernel that accepts the encoding and
the core's decoder can read it — that is how the converter measures error without knowing the method.

A quantiser's own scheme must be decoded by its `decode` hook (the oracle, `rad-kbench` and the
error report call it), and must return `RAD_E_UNSUPPORTED` for an encoding it did not write. Serving a
new encoding also needs a kernel library whose `layout` accepts it and an architecture that declares
the weight at it — a new format is a new kernel row plus a plugin, never a converter-only change.

Two plugins naming the same quantiser are refused, both named.

### The encoding model (`abi/rad_encoding.h`)

An encoding is the code type, an optional codebook, the grouping of scales/zero points/minimums, and
a transform. It is **data, not an enum**, so a plugin can invent one. The core can size, slice and
place any encoding from the struct alone, and decodes the `plain` and `affine` schemes itself:

```
q  = code                                   codes TILED, block [1x1], or [1xV] with a grid
q  = grid[code][k]                          grid TABLE [entries x V]
q  = table[code]                            table TABLE [1 x entries]
q  = -q if the sign bit is set              signs TILED u1
q  = q - zero                               zero TILED
v  = q * scale * scale.1 * ...              every scale level
v  = v - min * min.1 * ...                  a minimum in the value's domain
w  = v . T^-1                               the transform, per row over columns
```

So NVFP4 is fp4 codes, e4m3 scale per 16, f32 `scale.1` per tensor; Q4_K is u4 codes, u6 scale and
u6 min per 32 each times an f16 per 256; IQ2_XXS is u8 codes per 8 into a 256×8 grid, a sign bit per
value and two scale levels.

- Planes are **TILED** (cover the weight, element `(i,j)` covering a `[br,bc]` block; `0` extent means
  the whole extent) or **TABLE** (a codebook, grid, rotation parameter; every slice carries it).
- Sub-byte elements are a bit stream along the row, lowest bits first; every row starts on a byte.
- Transforms: `""`, `fwhtN` (Hadamard before quantisation; the `1/N` lives in the stored scale),
  `perm` (GPTQ act-order, a TABLE `t.perm`), `givens` (ParoQuant rotations: `t.pairs`, `t.angle`,
  optional `t.scale`, per-part `t.part`, composed `t.dense`).

A kernel only *rearranges* planes (`relayout`); choosing numbers is the quantiser's job. That split
is why a container never has to be reconverted when a kernel library changes its layout.

## Build, install, search path

```cmake
cmake_minimum_required(VERSION 3.21)
project(mykernels LANGUAGES CXX HIP)     # CXX alone for a host-only plugin
find_package(radiance REQUIRED)
rad_plugin_defaults()                    # C++20, optimising build unless asked otherwise
rad_add_plugin(mykernels kernels SOURCES rows.cpp HIP_SOURCES kernels.hip)
```

The kind is `kernels`, `architectures` or `quantizers` — the directory the engine loads it from.
Only `HIP_SOURCES` compile as HIP: for `RAD_GPU_TARGETS` or a `GPU_TARGETS` list, with `-DNDEBUG`
outside Debug, uncompressed, against the HIP runtime the installed radiance found. `find_package`
also sets `RADIANCE_ABI_VERSION` (to fail at configure time rather than at `dlopen`),
`RADIANCE_HOME`, `RADIANCE_KERNELS_DIR`, `RADIANCE_ARCHITECTURES_DIR`,
`RADIANCE_QUANTIZERS_DIR`. `cmake --install` puts the plugin under the home of the radiance you built
against; `RAD_PLUGIN_INSTALL_HOME` overrides that.

**`$RADIANCE_HOME` is a search path**: homes separated by `:`, the first winning; each may hold
`kernels/`, `architectures/`, `quantizers/`, and an earlier home's same-name file shadows a later
one. Your plugins go ahead of an installation without copying it:

```sh
RADIANCE_HOME=$HOME/radiance-plugins:/opt/radiance/share/radiance radiance --model m.rad
# in the container:
-v /my/plugins:/plugins -e RADIANCE_HOME=/plugins:/opt/radiance/share/radiance
```

The **kernel hierarchy** is `--kernels` (or `$RADIANCE_KERNELS`): `--kernels mykernels,libr4d` puts
yours before libr4d. An entry matches a plugin's name or its file (`libr4d`, `r4d`, `libr4d.so`, a
path). An entry matching nothing is warned about by name. With no list, every library found loads,
reference last.

## A library for a new card

The engine's own device code is compiled for every AMD family's generic target
(`gfx9-generic`, `gfx9-4-generic`, `gfx10-1-generic`, `gfx10-3-generic`, `gfx11-generic`,
`gfx12-generic`) beside the build's targets, so one engine binary starts anywhere. What it computes
there depends on the installed kernel library.

- Configure with `-DRAD_GPU_TARGETS=gfx942` (or `gfx11-generic`). At load the engine asks the runtime
  which targets the card accepts, in order (chip, then family: `gfx1036`, then `gfx10-3-generic`),
  with the card's modes (`gfx942:sramecc+:xnack-`), and takes the best fit in the fat binary.
- **A library with nothing for this card is left out, not failed**: its device rows leave selection,
  its host rows stay, and the log says what it carries versus what the card runs. One installation
  can carry a library per card. `rad_plugin_open` may decline explicitly with `RAD_E_UNSUPPORTED`.
- The engine uses discrete cards if any exist (all of the first one's architecture), integrated only
  when there is nothing else; `ROCR_VISIBLE_DEVICES` chooses explicitly. On integrated graphics,
  device memory is host memory and the budget leaves an eighth of host RAM to the host.
- **fp8 bytes are OCP E4M3** (`e4m3fn`, bias 7, max 448) wherever the engine reads them: container
  weights, an fp8 KV cache, activations. A card whose fp8 is `e4m3fnuz` (gfx942) converts in its
  `relayout` hook (the codes keep their bits except `0x80` and decode to half the value, so the scale
  doubles) and keeps activations/KV in OCP bytes, or converts at its own boundaries.
- **Tensor parallelism is ops you supply.** `all_reduce`, `all_gather` and the fused collectives are
  rows like any other; the engine inserts none and assumes no link.

## Checking your work

```sh
rad-kbench --kernels mylib,libr4d,libref --report mylib.md --bench
```

- It replays a recorded fixture of `libref`'s answers over real model geometries
  (`$RADIANCE_HOME/kernels.rkb`, plus `kernels_moe.rkb`), running every library with a row for a
  case. Correctness uses sampled values **and** the full-tensor norm **and** seeded projections;
  completeness is the op-by-library matrix; speed is timed on the same bytes.
- Red zones: every device allocation carries a poisoned 256-byte margin, verified after the launch.
- Fusion: a row with `replaces` is run against its named chain, out of your library, compared with
  `memcmp`.
- A skip is named, not failed — usually a missing `opd_shape`. `--strict` also fails a stale fixture.
- `rad-avx-check --ref libref.so --avx mylib.so` checks a **host** library without a card, op by op,
  at every ISA level and with parallel regions forced onto several threads (for a library exporting
  `avx_isa_force`/`avx_par_force`).
- `rad-schemas mylib.so libref.so` prints every schema disagreement at once (it exits 1 on any), and
  works on a plugin the engine cannot load.
- `--debug-selection` prints which row took each band and which constraint refused each other
  candidate; `/graph` on a running server shows the declared graph and which declarations the run
  phase issued.
- `rad-info --plugins` lists every plugin on the search path as the engine loads it: kind, version,
  declarations, reference status, whether its device code runs on this card, and its file.

## A worked example to copy

`plugin/spechub/` is the smallest real kernel plugin in the tree: two ops (`mtp_couple_propose`,
`mtp_couple_verify`), each with a device row and a host reference row. Read it in this order:

- `exports.cpp` — the whole export block: info, ABI, schema count/at, kernel count/at, plus
  `rad_kernel_reference` classifying device rows 0 and host rows 1 so the reference gate does not
  refuse the device rows.
- `spechub.cpp` — the rows, their constraints and their argument readers.
- `spechub.hip` / `ref_spechub.cpp` — the device launch and the host oracle.
- `CMakeLists.txt` — the `rad_add_plugin` call, the optional `enable_language(HIP)`, and
  `-ffp-contract=off` because an FMA in the ordered FP64 mass sums would move a token off the host
  reference.

## Related

- [[Radiance Architecture Plugins]] — the third plugin kind, one level up.
- [[Radiance Conversion]] — how a quantiser is driven by a recipe.
- [[Radiance Tools]] — `rad-kbench`, `rad-tune`, `rad-schemas`, `rad-avx-check` in full.
- [[Radiance Serve Options]] — `--kernels`, `--radiance-home`, the tuning aids.