diff --git a/serverAppConfigs-docs/LLM/Radiance Architecture Plugins.md b/serverAppConfigs-docs/LLM/Radiance Architecture Plugins.md new file mode 100644 index 0000000..983ce7b --- /dev/null +++ b/serverAppConfigs-docs/LLM/Radiance Architecture Plugins.md @@ -0,0 +1,287 @@ +--- +type: "howto" +title: "Radiance Architecture Plugins — declaring a model" +description: "Writing a radiance architecture plugin: declaring weights, buffers, KV groups and ops, then issuing the step." +tags: + - llm + - radiance + - plugins + - howto +timestamp: 2026-10-30 +--- + +# Radiance architecture plugins + +An architecture plugin declares a model in two phases: `rad_arch_declare` names every weight, +buffer, KV group and op **once**, against the model's metadata; `rad_arch_step` issues the ops for +one step. The engine picks the plugin by the exact `(rad_arch_id, rad_arch_quant)` pair — no +fallback — so one family at two weight formats is two `.so` files, and branching on a quant +descriptor inside one plugin is the anti-pattern the split exists to prevent. + +The in-tree plugins are the templates: `arch/llama_fp8/llama_fp8.cpp` (the smallest complete +decoder), `arch/gemma3`, `arch/qwen35_bf16`, and the header-only blocks in `arch/common/` +(`rad_block_attn*.h`, `rad_block_mlp*.h`, `rad_block_moe_fp8.h`, `rad_block_gdn*.h`, +`rad_block_dflash2.h`, `rad_block_mtp*.h`, `rad_block_vit.h`, `rad_block_ple.h`, +`rad_block_glm_*.h`, `rad_qsa.h`, …). Include those by `` — they are installed +beside the ABI and the in-tree plugins include them by that spelling. + +## Exports + +```c +const char* rad_arch_id(void); /* "llama", "qwen35", ... */ +const char* rad_arch_quant(void); /* "" unquantised, else the descriptor served */ +int rad_arch_serves_imports(void); /* optional: 0 declines an imported quantisation */ +int rad_arch_probe(const RadModelMeta* meta, RadArchProbe* out); /* optional */ +int rad_arch_declare(RadBuilder* b, const RadModelMeta* meta, const RadBuildCtx* ctx); +void rad_arch_step(struct RadCtx* c, const struct RadBatch* batch); +const RadChatFormat* rad_arch_chat_format(const RadModelMeta* meta); /* optional */ +const RadAudioFormat* rad_arch_audio_format(const RadModelMeta* meta); /* optional */ +``` + +Everything but declare/step is optional. `RAD_ARCH_PLUGIN(NS, ID, QUANT, VERSION, DESC)` in +`arch/common/rad_arch.h` emits the whole export block, including `rad_plugin_info` (kind +`RAD_PLUGIN_ARCH`). The plugin's **name must be unique**; the architecture id may repeat. + +`rad_arch_serves_imports` exists because a checkpoint whose quantisation was imported weight by +weight (Quark, ParoQuant, GPTQ, NVFP4) names its producing tool, which no plugin claims: the family's +one quantised plugin serves it, and two are refused by name. A second quantised plugin that reads +only its own containers returns 0. + +## Declare: the metadata + +`RadModelMeta` is what the loader read from the container's header before any plugin ran: `arch_id`, +`name`, `quant`, `n_layers`, `n_embd`, `n_head`, `n_head_kv`, **`head_dim`**, `n_ff`, `n_vocab`, +`n_ctx_train`, expert counts, `rms_eps`, `rope_theta`, and a free-form `kv_key`/`kv_val` table for +everything else (`rad_meta_geti/getf/gets`). Read dimensions from here — one plugin covers every size +in a family. + +**`head_dim` is read, never derived.** On the Qwen hybrids it is 256 while `hidden / heads` is 213; +a derived value silently breaks projection widths, KV page size and the attention kernel's band. + +`RadBuildCtx` is the deployment the plugin must declare against: + +| field | meaning | +|---|---| +| `rank`, `world_size` | declare runs once per rank; **dimensions are already divided** — the core inserts no collectives | +| `max_tok`, `max_seqs`, `max_ctx`, `max_spec` | the step bounds: buffer sizing, kernel bands, the scheduler's window | +| `max_mixed_tok` | the most a multi-sequence step may carry; a wider step is one sequence (prefill span) | +| `tp_wire_lossy`, `tp_wire_min_bytes` | invert into `all_reduce`'s `exact`; below the floor a message must go exact | +| `kv_dtype`, `kv_quant` | the attention cache width and, under fp8, its narrow layout (`RAD_KVQ_V4`, `RAD_KVQ_K6V4`) | +| `attn_prefill_fp8` | `RAD_ATTN_FP8_*` bits to pass as `attn_paged`'s `prefill_fp8` | +| `kv_resident_blocks`, `kv_stream_ok` | KV streaming: the ring depth, and permission to declare the streamed-only ops | +| `state_dtype` | `RAD_F16` when `--state-cache-dtype f16` is served by this plugin | +| `is_draft`, `scope` | a drafter/encoder declare; `scope` prefixes declared names (`draft`, `vision`, `audio`) | +| `shape_probe` | a **sizing** declare at a smaller `max_tok`; must not touch state the real declare wrote | +| `max_enc_patches`, `max_audio_frames` | the encoder pass sizes | +| `max_out_rows` | logits rows the KL mode needs (0 = the plugin's own rule) | +| `cvec` | the loaded control vector, or null | +| `tp_attn_rows` | row-split prefill attention requested | +| `draft_coupling`, `draft_tree` | declare the coupled/tree forms | +| `weight_prefetch_off` | declare no weight prefetch beside collectives | + +## Declare: weights + +```c +static const RadWeightDecl d = { + .dtype = RAD_BF16, .rank = 2, .shape = {n_embd, n_ff}, + .access = RAD_ACCESS_PER_TOKEN, .shard = RAD_SHARD_ROW, + .group = grp_layer(l), .optional = 0, + .n_row_parts = 0, .source = nullptr, .planes = nullptr, +}; +rad_weight w = rad_decl_weight(b, "blk.0.mlp.gate.weight", &d); +``` + +- **`access`** tells the placement planner how the weight is read: `PER_TOKEN` (dense), + `PER_REQUEST` (an encoder tower, read once per request), `CONDITIONAL` (routed experts), + `RARE`, `VOCAB` (lm_head/embedding: read every token but huge, and one is a gather). +- **`shard`**: `NONE` (replicated), `ROW` (split dim 0), `COL` (split dim 1, needs a following + all-reduce). The plugin says; the core does not infer. Uneven splits use + `rad_weight_shard_span`/`_parts` (a width that does not divide into whole 128-column blocks; the + delta net's `[q|k|v|z]` of unequal part widths). +- **`group`** is the unit of movement (`layer`, `expert`, `slot`); an expert's gate/up/down share one, + because the unit of transfer is an expert, not a tensor. +- **`row_parts`** describes what a `SHARD_ROW` weight stacks along dim 0 (`[gate|up]`, + `[q|k|v|z]`). **A tensor-parallel rank owns a slice of each stacked projection**, never a + contiguous slice of the stack; the parts cannot be inferred from a fused container tensor, and a + contiguous slice produces fluent wrong text. +- **`source`/`planes`** view an existing stored weight: a block-fp8 linear's `codes,scale` are one + logical weight taken by a GEMM as two operands. Null `source` is the weight of its own name; null + `planes` takes every plane. +- A dtype other than the plane's is legal only as an exact widening (a bf16 norm read as f32). + +The **name map** (`rad_decl_name_map`) is the plugin's map from checkpoint tensor names to declared +weight names, and it is the only place convert-time fusions live: `COPY`, `CONCAT`, `WNORM` (a +weight-normalised pair folded to the plain weight), `ROWS`, `TRANSPOSE`, `EMA` (an EnCodec codebook), +`TAP`, `COLS`. `rad-convert` reads this map from the plugin, so it needs no knowledge of the model. +`RAD_MAP_COPY` with several sources means "first present wins" (a tied lm_head); `CONCAT` requires +all. Ask for a weight's encoding with `rad_weight_encoding` **after** its map entry when the graph +depends on the format (fp8 GEMM vs plain, rotated expert kernels). + +## Declare: buffers and KV groups + +`rad_decl_buffer` names every activation: `RAD_BUF_TRANSIENT` (liveness-analysed, may share bytes), +`RAD_BUF_PERSIST` (across steps: a calibration accumulator, a routing histogram), or +`RAD_BUF_DERIVED` (computed once per step by the core from the batch — `derive` names the +derivation, e.g. `conv_state_index`). Buffers live in one arena sized once, not an allocator per +step. + +`rad_decl_kv_group` declares each state kind: `RAD_KV_FULL` (paged per-token, growing), +`RAD_KV_WINDOW` (bounded by a window), `RAD_KV_LINEAR` (per-sequence fixed state, GDN), +`RAD_KV_CONV` (a rolling window of `width-1+num_spec`), `RAD_KV_RING` (window floor set at runtime). +The **block size is not declared** — it comes from the resolved attention kernel's constraints; read +it back with `rad_kv_block_size`. `rad_bind_layer_kv` binds each layer so the block manager knows how +many state instances a sequence owns; `rad_kv_group_zero` (a rank with no heads), +`rad_kv_group_replicated` (every rank holds the same bytes), and `rad_kv_group_dense_read` (read over +the whole history every step, so it cannot stream) are the three refinements. + +## Declare: ops + +```c +rad_op h = RAD_OP(b, "gemm_w4a8_nt", + RAD_PARAMS(RAD_RANGE("M", 1, max_tok), RAD_INT("N", 8192), RAD_INT("K", 5120)), + RAD_WEIGHTS(w)); +rad_op_reads (b, h, RAD_BUFS(b_x), n); +rad_op_writes(b, h, RAD_BUFS(b_y), n); +``` + +- Parameter names and operand order come from `docs/OPS.md`; a new name is allowed only when the + vocabulary has none, and the first plugin to declare it fixes its schema. +- `RAD_RANGE` on exactly one parameter per op gives the selector its bucket table. The core carries + every parameter opaquely except `RAD_PROLE_SEQ_CHUNK` (cut steps on it) and `RAD_PROLE_CAPACITY`. +- `rad_op_reads`/`rad_op_writes` are the buffer planner's liveness. An undeclared use is assumed live + for the whole program: sound but wasteful. +- **Fusion is selection.** Ask for the fused op; if `rad_op_resolved` is false, emit the chain. A + fusion's `replaces` hook is a checkable equivalence, not a rewrite. +- `rad_op_declared(b, "op")` asks whether any library declares an optional op; use it before + emitting a fused form only an optional library defines. +- Buffers touched from the **second lane** go through `rad_buf_concurrent`, and an architecture whose + lane-1 work lies in one stretch declares `rad_lane1_window_begin/end` around it. The runtime + refuses a lane-1 op outside the window; the window lets buffers used only outside it share bytes. + +Step output: `rad_declare_logits` names the buffer the sampler reads (its dim 0 bounds `n_out`; a +[n_tok, vocab] logits buffer at a large vocabulary is gigabytes — declare the gather + +`logits_gemm` shape instead). An embedding program declares `rad_declare_pooled` instead (one row a +sequence, no sampler, no generation). `rad_declare_scores` opts into host-side scored steps. + +## Declare: drafters, encoders, generators + +- `rad_declare_drafter(RadDrafterDecl)` — depth, block, proposal shape. `rad_drafter_shared_kv` (a + drafter with no cache of its own), `rad_drafter_tree` (a tree over the proposals), + `rad_drafter_short_block` (a second, shorter pass) are refinements. Declare the drafter **last**. +- `rad_declare_encoder(RadEncoderDecl)` — a vision/audio tower, media preprocessed by the core. +- `rad_declare_generator` / `rad_declare_audio_generator` / `rad_declare_video_generator` — image, + audio and video generation programs; the engine sizes the step from the probe's `gen_*` fields + before declare. +- Optional format exports let a model state what it knows and the server learn nothing new: + `rad_arch_chat_format` (a data description of reasoning/tool-call markup, replacing template + analysis), `rad_arch_audio_format` (how a speech model hears: clip tokens, prompt, language, + transcript start), `rad_arch_text_prepare` (rewrite request text into training form), + `rad_arch_video_positions`/`rad_arch_video_pack` (video pass geometry and packing), + `rad_arch_encode_geom` (encoder pass geometry). Each struct grows only at its end behind + `struct_size`, so adding a field does not move the ABI. + +**Media preprocessing is the core's**, not the plugin's: resizing, patching, placeholder expansion +and positions in six processor families (Qwen-VL, Gemma 4, LFM2-VL, Unlimited-OCR, GLM-5.3, Pixtral +for Nemotron), chosen by `vision_config.model_type` plus owning-model checks; audio through Gemma 4's +extractor or Qwen3-ASR's Whisper one, chosen by `audio_config.model_type`. A vision model with another +pipeline needs a core change. + +## Step: issuing the graph + +`RadBatch` is built once per step and shared by every block: `phase` (`PREFILL`/`DECODE`/`MIXED`), +`n_tok`, `n_seq`, `token_ids`, `positions`, `cu_seqlens`, `seq_ids`, `lora_slot`, `n_out`/`out_ids`, +`q_lens`/`ctx_lens`/`max_q_len`/`max_ctx_len`, the KV-group array, the speculative fields +(`n_spec`, `draft_pass`, …). + +```c +static void step(RadCtx* c, const RadBatch* batch) { + if (batch->draft_pass > 0) { m.ds.step(c, batch); return; } /* query pass */ + if (batch->draft_pass < 0) { m.ds.context(c, batch); return; } /* context pass */ + RAD_ISSUE(c, m.op_embed, praw(batch->token_ids, RAD_I32, batch->n_tok), + RAD_W(m.w_tok), brows(m.b_x, batch->n_tok)); + for (auto& layer : m.layers) { layer.attn.step(c, batch); layer.mlp.step(c, batch); } + /* ...heads; the lm_head only over batch->n_out rows... */ +} +``` + +- `rad_issue(c, op, operands…, n)` / `RAD_ISSUE` (n = the batch's ranged extent) / + `RAD_ISSUE_N` (an explicit row count). `rad_step_fail(c, what)` sets the status helpers return. +- **A mixed step is the trap.** Continuous batching produces steps with a prefill chunk beside decode + rows. A block that issues one path for the whole step sends its decode rows down the chunked path, + which commits recurrent state directly and takes no `num_accepted`; a rejected draft cannot be + rolled back, and the model stays fluent while drifting. Issue the recurrent path over + `RadBatch::n_seq_decode` sequences and the chunked path over the rest (the core orders steps + decode-first, so each half is contiguous; `cu_seqlens` holds absolute offsets, so pass + `cu + n_seq_decode` with the token operands at their base). +- A step issued on lane 1 outside the declared window fails by name. An op the core appends behind a + declared one (runtime LoRA) counts at that op's position. +- **Zero rows is not a call**: issuing an op over zero rows (a zero ranged extent, or all-empty + sequences) returns `RAD_E_SHAPE` and writes nothing. The caller skips the issue, as the + architectures' decode and prefill halves do. +- `n_out` is 0 for a chunk that finishes no sequence — do not run the lm_head. + +## Probe: what the plugin measured + +`rad_arch_probe(meta, out)` runs before declare. Its fields are the engine's permissions: + +| field | effect | +|---|---| +| `draft_depth`, `draft_depth_fixed` | the container's spec depth; a block drafter's trained block is the only depth it serves | +| `shape_probe_ok` | the plugin tolerates a sizing declare, so the activation arena can be lent | +| `enc_merge`, `audio_frames` | encoder pass granularity; longest audio clip | +| `gen_rows`, `gen_patches`, `gen_audio`, `gen_ctx`, `gen_video` | generator sizing before declare | +| `kv_quant_ok` | bits of the narrow KV layouts this model's kernels serve | +| `chain_pays`, `tree_pays` | the plugin measured chained launches / draft trees faster | +| `kv_stream_ok` | the architecture can stream a KV group (sparse, block-addressed, with `kv_resolve`) | +| `control_vector_ok` | the plugin applies `RadBuildCtx::cvec` | +| `tp_attn_rows_ok` | the plugin serves row-split prefill attention | +| `state_f16_ok` | the plugin declares recurrent state at f16 and its kernels read it | + +A flag whose probe bit is 0 is **refused at startup with the plugin named** (`--kv-cache-dtype +fp8_v4`, `--control-vector`, `--tp-attn-split rows`, `--state-cache-dtype f16`, `--kv-resident`). +That is the design: a deployment switch the plugin cannot honour is an error, never a silent +substitution. + +## Worked pattern: the smallest complete plugin + +`arch/llama_fp8/llama_fp8.cpp` builds twice from one source — `llama_fp8` (quant `fp8_e4m3`) and +`llama_dense_fp8` (quant `""`), differing only in the descriptor and the graph-dump text. It shows +the whole shape: + +1. `struct Layer { AttnFP8 attn; MlpFP8 mlp; }`, `struct Model` holding `Names`, `Geom`, buffer and + op handles. +2. `declare()`: name map entries first, then weights (with `rad_weight_encoding` where the head's + format decides the kernel), buffers, KV group, the embedding and its all-reduce, then each layer's + `declare()` from the shared blocks, the final norm, gather and lm_head, **the drafter last**, and + `rad_note` lines explaining the shape. +3. `step()`: draft passes first (sign selects query/context), then embedding (+ exact all-reduce + when needed), each layer's `step()`, the final norm, and the head only over `n_out`. +4. `probe()`: reads `draft.block_size` from the metadata and refuses any other depth for a block + drafter. +5. `RAD_ARCH_PLUGIN(...)` at the bottom. + +Add a new family by copying a neighbour, replacing the name map and geometry, and composing the +blocks you need; add a new *format* by cloning the plugin with another `(id, quant)` pair and the +ops that format needs. + +## Verifying a new plugin + +```sh +radiance --model m.rad --debug-graph # every op, its resolved kernel, and why +radiance --model m.rad --debug-placement # the plan +rad-convert $SRC -o m.rad --plan-only -v # a convert that plans will boot +rad-kbench -m m.rad --kernels mylib,libr4d,libref --report k.md --bench +rad-tune -m m.rad --tp 2 # only at the shapes you serve +``` + +`tests/arch_test.cpp` drives a plugin's builder the way the core does, including lane-window +liveness, so the pinned graphs and steps are the ones the core plans. A plugin that declares a weight +form no kernel `layout` accepts is a declare-time error naming the weight, encoding and kernel — the +same check `rad-convert` runs before writing. + +## Related + +- [[Radiance Plugins]] — kernel libraries, encodings, the build path, ABI law. +- [[Radiance Conversion]] — how the name map drives a convert, and `--set`/descriptor selection. +- [[Radiance Model Families]] — the existing `(id, quant)` pairs and their recipes. +- [[Radiance Tools]] — `rad-kbench`, `rad-tune`, `rad-info --plugins`. diff --git a/serverAppConfigs-docs/LLM/Radiance Conversion.md b/serverAppConfigs-docs/LLM/Radiance Conversion.md new file mode 100644 index 0000000..f95b0c2 --- /dev/null +++ b/serverAppConfigs-docs/LLM/Radiance Conversion.md @@ -0,0 +1,354 @@ +--- +type: "howto" +title: "Radiance Conversion — checkpoints into .rad containers" +description: "Turning a checkpoint into a .rad container: rad-convert, the recipe language, quantisers, reuse and export." +tags: + - llm + - radiance + - quantization + - howto +timestamp: 2026-10-30 +--- + +# Radiance conversion + +`rad-convert` turns a Hugging Face checkpoint into a `.rad` container, choosing each weight's +encoding from a **recipe**. `rad-export` wraps it for finetunes (a merged checkpoint or a PEFT +adapter) with the stock model's recipe. Neither opens a GPU: conversion is CPU-only and safe beside a +running server. + +Fetch checkpoints with `scripts/hfget.sh ` (resumable; reads `.safetensors`, +skips `.bin`/ONNX/GGUF when safetensors exist; `HF_TOKEN` for gated repos; writes a +`.hfget-manifest` of content keys). `radiance --model ` runs the same fetch itself. + +## What a conversion does + +1. Read `config.json` (or GGUF metadata), and the shard index if present. +2. Load the architecture, kernel and quantiser plugins and run **declare**, as the engine does. +3. Resolve every op and ask each resolved kernel's `layout` hook about the planned encoding. A + recipe no kernel can serve is refused **here**, before any I/O. +4. Write each logical weight once: the checkpoint's bytes where no rule matches, otherwise the + planes the matching quantiser writes, a block of rows at a time. +5. Measure every quantised weight against its source through the core's decoder and print the worst. +6. Bake the tokeniser, chat template, metadata, recipe and any draft head. + +```sh +build/bin/rad-convert /srv/models/hf/Qwen/Qwen3.8-27B -o qwen3.8-27b-bf16.rad \ + --recipe data/recipes/q38-27b-bf16.recipe +build/bin/rad-convert /srv/models/hf/Qwen/Qwen3.8-Flash-Next -o flashnext-bf16.rad --max-ctx 4096 +``` + +## Every `rad-convert` option + +| option | meaning | +|---|---| +| `` | a `.gguf`, a `.safetensors`, an `index.json`, or a checkpoint directory | +| `-o, --out FILE` | the container to write | +| `--recipe FILE` | one rule a line: `PATTERN QUANTISER key=value …`; repeatable; first match decides | +| `--quant RULE` | `PATTERN=QUANTISER[:key=value,…]`, placed **ahead of** the recipe's rules; repeatable | +| `--list-quantizers` | every loaded quantiser and the options it takes, then exit | +| `--arch ID` | override the architecture id read from the checkpoint | +| `--home DIR` | `$RADIANCE_HOME` (kernels, architectures, quantizers) | +| `--kernels A:B` | kernel libraries to load, first wins; declare checks the recipe against them | +| `--tokenizer F` | `tokenizer.json`; its declared chain is baked instead of reconstructed from a pre-tokenizer name. Required for a GGUF, without it the chain is a guess | +| `--tokenizer2 F` | a second `tokenizer.json` baked as the second vocab (a generator reading its prompt through two tokenisers); keep `tokenizer_config.json` beside it | +| `--add FILE` | merge one more `.safetensors`, tensor names kept; repeatable; a name in two files is refused | +| `--config DIR` | read `config.json`, `generation_config.json`, `tokenizer.json` from DIR for a single-file checkpoint | +| `--chat-template F` | a Jinja template baked in, overriding the checkpoint's (or its absence); trailing newlines stripped; without one `/v1/chat/completions` answers 501 | +| `--imatrix F` | llama.cpp imatrix: expert-popularity profile, per-column importances, error weighting; required by `ggml` types `iq2_xxs`, `iq2_xs`, `iq1_s` | +| `--set K=V` | a metadata key, seen by declare **and** written into the container; repeatable | +| `--max-tok N` | the `max_tok` declare runs at (default 8192, or a generator's largest pass) | +| `--max-ctx N` | the context declare runs at (default: the checkpoint's training context) | +| `--max-spec N` | the `max_spec` declare runs at (default 4, or the block a drafter states); `0` leaves a draft head out | +| `--draft-model DIR` | a drafter shipped as its own repository, merged under `draft.` | +| `--kv-cache-dtype T` | `bf16` or `fp8`: the cache width declare runs at; the container is identical at either | +| `--reuse FILE` | copy every weight an earlier container holds with the same encoding, quantiser, options and shape | +| `--changed FILE` | with `--reuse`: checkpoint tensors (one name a line) that differ; every weight reading one is converted | +| `--in-place` | with `--reuse`: extend that container; new weights go past its end, the header last; keeps `.pre-append` | +| `--lora FILE\|DIR` | merge a PEFT adapter into the checkpoint tensors before converting (`W + strength · scale · B·A`); repeatable, merged in order; each merged tensor joins `--changed` | +| `--lora-strength F` | strength for every `--lora` (default 1) | +| `--lora-map FILE` | JSON map for adapters whose module names differ: `{"strip":[prefix…],"modules":{"":[target…]}}`; one `*` matches a block index; several targets mean a fused module (rows split in order) | +| `--plan-only` | declare, plan every encoding, print the plan, stop before writing | +| `--report N` | print the N worst quantisation errors (default 20) | +| `-v, --verbose` | per-weight detail | + +`--reuse` matches a weight on its **encoding, quantiser, options and shape — not on its source +bytes** — and carries whatever calibration made the old planes. Use it only when source and +calibration are unchanged, or name the changed tensors with `--changed`. `--in-place` refuses to +change a weight the container holds or to add an expert. + +## The recipe language + +``` +# PATTERN QUANTISER OPTIONS +blk.*.ffn_*_exps.*.weight gptq codes=i4 group=128 scale=bf16 transform=fwht128 calib=$CALIB +blk.*.ple_ngram.weight rtn codes=fp8_e4m3 block=*x* scale=bf16 rule=fixed scale_value=1.99317932128906e-4 +output.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16 +blk.*.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16 +``` + +- One rule a line: `PATTERN QUANTISER [key=value …]`. `#` starts a comment. +- `PATTERN` is a glob over the **whole logical weight name**: `*` crosses dots, `?` is one character. +- **The first matching rule decides.** No rule → the checkpoint's own encoding, which must be one + the engine serves as is: `bf16`, `f16`, `f32`, or block-scaled FP8 (`X.weight` with + `X.weight_scale_inv`). Anything else is named and refused. +- `$NAME` / `${NAME}` in a value comes from the environment; an **unset variable is refused**, never + expanded empty (so an empty `$CALIB` cannot silently produce an uncalibrated convert). +- A quantiser may **decline** a weight (`RAD_E_UNSUPPORTED`), which keeps the checkpoint's encoding + — `rtn` declines a rank-1 tensor, so a pattern that also matches a norm leaves the norm alone. +- An option the quantiser does not declare is refused, by name, before anything is quantised. +- Two plugins naming the same quantiser are refused, both named. + +## Quantisers + +| quantiser | what it does | +|---|---| +| `rtn` | round to nearest on a grid; each block's scale chosen by `rule` | +| `gptq` | error feedback against the layer's input Hessian (`calib=DIR`, damped by `damp` = 0.01 of the mean diagonal by default); `act_order=1` quantises columns by descending input energy and stores the permutation; `cd=N` adds coordinate-descent sweeps | +| `awq` | per-input-column scaling to a searched power, then a per-block clip search, both against the Hessian; column scales are the last scale level | +| `autoround` | sign-gradient descent on a rounding offset per value and a range factor per block; `iters` steps; slow (a 27B dense model takes most of a working day at defaults; `iters=50` a quarter) | +| `paroquant` | channel scaling plus `stages` Givens rotations learned against the Hessian, then AutoRound on the rotated weight; the `givens` transform | +| `m2xfp4` | M2XFP's weight-side Sg-EM: MXFP4 plus one metadata byte per group of 32; its own `m2xfp4` scheme; `adaptive=0` pins the bias, `wide=1` widens it | +| `ggml` | a llama.cpp format (`q4_0`, `q4_1`, `q5_0`, `q5_1`, `q8_0`, `q2_K`…`q6_K`, `iq4_nl`, `iq4_xs`, `iq2_xxs`, `iq2_xs`, `iq2_s`, `iq3_xxs`, `iq3_s`, `iq1_s`, `iq1_m`), by llama.cpp's own search, imatrix-weighted, codes byte for byte | +| `cast` | a float weight in another float dtype (`dtype=` f32, bf16, f16, fp8_e4m3, fp8_e5m2), no scale | +| `import` | rad-convert's own, not a plugin: write the numbers a quantised release ships, in the spelling the kernels read, instead of quantising again | + +Every quantiser except `m2xfp4`, `ggml`, `cast` and `import` takes the grid options: + +| option | values | +|---|---| +| `codes` | `i2`…`i8`, `i16`, `i32` symmetric; `u1`…`u8`, `u16` with `zero=`; `fp8_e4m3`, `fp8_e5m2`, `fp6_e2m3`, `fp6_e3m2`, `fp4_e2m1` | +| `table` | a codebook the codes index: `nf4`, `iq4nl`, `w4nl`, `w5nl`, `binary`, `ternary`, or `;`-separated values | +| `group=N` / `block=RxC` | where the scales sit; `*` is a whole extent | +| `scale` | f32, bf16, f16, fp8_e4m3, e8m0; under `scale2` also an integer sub-scale `u4`…`u8`, `i6`, `i8` | +| `scale2`, `group2` / `block2` | a second scale level the first is relative to (default the whole tensor) | +| `scale2_value` | the second level fixed at this value, exact in `scale2`'s dtype | +| `zero` | an integer zero point per block — an asymmetric grid | +| `transform=fwhtN` | rotate each row by the Hadamard first | +| `rule` | `absmax`, `mx` (OCP shared exponent), `fixed` (`scale_value=`), `search` (least squared error over candidates and refits, imatrix-weighted) | +| `clamp` | a signed code's range: `full` or `sym` | + +Named grids: MXFP4 = `codes=fp4_e2m1 group=32 scale=e8m0`; NVFP4 = +`codes=fp4_e2m1 group=16 scale=fp8_e4m3 scale2=f32`; NF4 = `table=nf4 group=64`; a K-quant-like grid += `codes=i4 group=32 scale=u6 scale2=f16 group2=256`; a ternary BitNet weight = `table=ternary`. + +**A container can hold only what some kernel reads.** rad-convert checks every planned encoding +against the `layout` hooks of the kernels declare resolved and refuses one no kernel accepts. libr4d +reads the plain floats; block-scaled fp8 and fp8 rows; int8 block rows plain or rotated; routed +experts as fp8, int4 (`i4*bf16[1x128]/fwht128`) or the `w4nl`/`w5nl` codebooks under E4M3 scales; +the dense w4, w8 and mxfp4 GEMM rows; and the 2-bit draft head. See [[Radiance Plugins]] for the +encoding model. + +## Importing a release someone else quantised + +These forms are read weight by weight and written in the spelling the kernels read; a recipe rule +re-quantises from the exact dequantisation where the family's kernels want another form. + +| `quantization_config` | read as | notes | +|---|---|---| +| `quark` MXFP4 (fp4, group 32, e8m0) | `fp4_e2m1*e8m0[1x32]` | byte for byte; also servable as it stands | +| `paroquant_mxfp4` | `fp4_e2m1*e8m0[1x32]/givens` | codes/exponents byte for byte plus learned rotations as `givens` tables | +| `paroquant`, `int5-bitplane`, plain `paroquant` | `u5*f16[1x128]-u8[1x128]/givens`, `u4*…` | AWQ-packed nibbles re-laid as a code bit stream; f16 scales, integer zeros, rotations | +| `compressed-tensors` `pack-quantized`, 4-bit `int`, group, act-order none/static/weight | `i4*[1xG]` | `weight_packed` made signed beside `weight_scale`; group/dynamic reorder is refused | +| same with `symmetric:false` (AWQ) | `u4*[1xG]-u4[1xG]` | nibbles plus `weight_zero_point`, bit for bit | +| `auto-round` `auto_round:auto_gptq`, `gptq` v1 | `i4*bf16[1x128]` or `i8*bf16[1x128]` | scales rounded to bf16; convert-time; `import` keeps shipped f16 | +| `nvfp4-pack-quantized` | `fp4_e2m1*f32[1x16]` | `weight_scale / weight_global_scale`; **no kernel serves NVFP4**, so it always takes a rule | +| `modelopt` NVFP4 / FP8 / MIXED_PRECISION | NVFP4 `fp4_e2m1*f32[1x16]`; FP8 `fp8_e4m3*f32[1x*]` | ModelOpt's second scale **multiplies**; `input_scale` is not read | +| compressed-tensors all-FP8 groups | `fp8_e4m3*[1x*]` or `[RxC]` | nothing imported: stock FP8 with its scale; a per-tensor scale is written once a row | +| GGUF `Q8_0`, `Q4_K`, `Q6_K` | `i8*f16[1x32]`, affine u4/u6…, `i6*i8…` | blocks split bit for bit; **not servable as they stand** | + +The width is read **per tensor**, so a mixed release imports int4 experts and int8 projections +together. What an import cannot read (an asymmetric GPTQ zero point, a group-reordering act-order +index, a GPTQ group under 128, a missing/zero global scale, an unlisted format) is refused by name. +The release's `quantization_config` names the tool, not a served form, so it is kept as +`imported_quantization_config.*`. + +`import` options: `codes=uN` makes a symmetric grid unsigned under a zero point of `2^(N-1)`; +`zero=` the zero plane's dtype (`u4`, `u8`); `scale=` f16/bf16/f32, each shipped scale rounded to +nearest; `block=RxC` repeats an FP8 scale shipped per row/tensor over that grid. It reads an imported +integer grid or FP8 with a per-row/per-tensor scale and keeps a plain float weight as it is. + +**NVFP4 to MXFP4 rounds twice** (e4m3 per 16 → power of two per 32), so it loses more than MXFP4 made +from bf16. Prefer a calibrated MXFP4 container when one exists; the quality verdict is a KL gate, not +a file size. + +## The descriptor law (`--set`) + +The architecture plugin is chosen by the exact `(arch id, quant descriptor)` pair. A convert-time +choice that must also hold at serve time is stated once with `--set`, which hands it to declare and +writes it into the container. Storage format is not such a choice (the encoding is in the container +and the plugin reads it there), but the **serving plugin** is: a `quantization_config.*` key +re-derives the descriptor. + +```sh +rad-convert -o q38-27b-fp8.rad \ + --recipe data/recipes/q38-27b-fp8block-df2.recipe \ + --set quantization_config.quant_method=fp8 --set quantization_config.fmt=e4m3 \ + --set "quantization_config.weight_block_size=128 128" \ + --set quantization_config.activation_scheme=dynamic +``` + +A bf16 checkpoint whose recipe writes Qwen's block FP8 must therefore be declared *as that format*; +otherwise the selection pair does not match any plugin and the convert (or the boot) is refused. The +same law bites the other way when a release's own `quant_method` names a pair no plugin claims. + +## Inspect and validate the result + +```sh +rad-info out.rad # architecture, source quant, recipe, sections +rad-info --recipe out.rad # the expanded rules — did the intended one win? +rad-info -v out.rad | head -40 # every tensor, its encoding, and what made it +rad-kbench -m out.rad --home $RADIANCE_HOME --report kbench.md +radiance --model out.rad --debug-graph # every op resolves +``` + +The converter's own error report (the worst `--report N` weights) measures each quantised weight +against its source through the core decoder. `rad-kbench` adds the artifact test: weight operands +read from the container and passed through each kernel's `layout`/`relayout` hooks. Both are on +[[Radiance Tools]]. + +## Calibration (GPTQ, AWQ, AutoRound, ParoQuant) + +`calib=DIR` is a directory an engine run wrote with `RADIANCE_CALIB_DIR` set: each layer's **Gram +matrices of its input**, in the domain the kernel multiplies it in. A block that characterises an +activation declares an f32 `[K,K]` PERSIST buffer and a `gram_accum` op; at `RADIANCE_CALIB_TOKENS` +(default 65536) the run writes the files. + +Three-step bootstrap: convert with plain `rtn` → serve that container with `RADIANCE_CALIB_DIR` → +push real prompts through prefill → convert again with the Hessians. The second moment is robust to +the first container's small quantisation error. A weight with no file is quantised uncalibrated and +the converter says so; `awq`, `autoround` and `paroquant` fall back to the imatrix diagonal when one +is given. + +- The accumulator must be **PERSIST**: a transient buffer may share bytes and be overwritten between + the launches it sums. +- Files are per rank; the consumer sums them. Under expert parallelism a rank's `down` input covers + only its experts' tokens; `gate_up` input is the full token set on every rank, so the sum scales + `H` by world size — GPTQ is invariant to a positive scale. +- Under tensor parallelism a column-split linear sees only its rank's columns, so a tap may write + `gram.r..part.bin`; libquant places the parts on the diagonal in rank order, leaving + cross-rank products zero. They must cover K and be in the model's own domain. +- `--imatrix` supplies the expert-popularity profile the placement planner ranks units by, the + per-column importances a quantiser uses, and the error report's weighting. + +## Reuse, drafters, merging + +```sh +# re-quantise only what changed +rad-convert $SRC -o new.rad --recipe new.recipe --reuse old.rad --changed changed.txt + +# merge a drafter shipped as its own repo (DFlash2, DSpark, MTP assistant) +rad-convert $SRC -o out.rad --draft-model /srv/models/z-lab/Qwen3.8-27B-DFlash2 \ + --recipe data/recipes/q38-27b-mxfp4-df2.recipe + +# merge a PEFT adapter at convert time +rad-convert $SRC -o tuned.rad --lora ~/run/adapter --lora-strength 1.0 \ + --recipe data/recipes/q38-27b-mxfp4-df2.recipe +``` + +- A drafter's tensors go in under `draft.`, its config under `draft.` keys. It has no embedding and + no `lm_head` of its own; it uses the target's. +- `--lora-map` exists for adapters trained on names the checkpoint does not use; a fused target + (`qkv`) takes several targets and splits the adapter's rows in order. +- `--add` + `--config` + `--tokenizer2` assemble a ComfyUI-style release of separate single files + (Anima: diffusion model, text encoder, VAE). A name in two files is refused. + +## `rad-export` — finetunes without hand-writing the command + +```sh +rad-export ~/outputs/my-run/checkpoint-400 --dry-run # what it would run +rad-export ~/outputs/my-run/checkpoint-400 # adapter -> .rad +rad-export ~/exports/merged -o ~/models/mine.rad --recipe q38-27b-fp8block-df2 +``` + +| flag | default | meaning | +|---|---|---| +| `SRC` | required | a merged 16-bit checkpoint (`config.json` + `*.safetensors`) or a PEFT adapter (`adapter_config.json` + `adapter_model.safetensors`) | +| `-o, --out FILE` | SRC's folder name + `.rad`, beside SRC or in `out_dir` | the container | +| `--recipe NAME\|FILE\|auto` | `auto` | a recipe name, a file, or the one recipe whose header names the base model, takes a bf16 checkpoint and needs no calibration | +| `--base DIR` | the adapter's `base_model_name_or_path`, from disk or the HF cache, else downloaded | the base a LoRA merges into | +| `--draft DIR\|ORG/NAME` | the config, else the Hub repo the recipe's usage line names | the drafter when the recipe merges one; a usage line with a placeholder asks for `--draft` | +| `--image IMAGE` | `$RADIANCE_IMAGE`, the config's, then `stilldeadcode/radiance` | run `rad-convert` in this image (docker, CPU-only, no network, **not pulled**) | +| `--rad-convert PATH` | — | run this binary instead of docker; recipes come from its `share/radiance` | +| `--memory M` | — | docker memory limit for the convert | +| `--dry-run` | — | resolve recipe, drafter and paths; print the `rad-convert` line; write nothing | +| `--keep-merged` | — | keep the merged checkpoint beside OUT | +| `--config FILE` | `~/.config/rad-export/config.json` | the config below | +| `--list-recipes [--json]` | — | list the recipes the image/installed tool carries | + +- Input: a merged 16-bit checkpoint (`config.json` + `*.safetensors`) or a PEFT LoRA adapter + (`adapter_config.json` + `adapter_model.safetensors`). +- An adapter is merged **exactly as PEFT's `merge_and_unload`** computes it on CPU: `scale` = + alpha/r (alpha/√r with rsLoRA), `rank_pattern`/`alpha_pattern` honoured, delta in fp32 rounded to + the adapter's dtype. It streams tensor by tensor into 1 GiB shards, so a 27B merge needs a few GiB + of RAM, not the model twice. Plain linear LoRA only: embedding LoRA, trained biases, DoRA, saved + modules, convolution LoRA, a rank other than the config's, and a missing target are refused before + anything is written. The adapter's tokeniser/template replace the base's where shipped. +- The recipe comes from the image (`/opt/radiance/share/radiance/recipes`) or the installed + `rad-convert`'s share; `auto` matches the base model's name in recipe headers and asks when several + fit. A recipe needing calibration is listed but never picked by `auto` — a finetune's Hessians are + not the stock model's. +- It runs `rad-convert` on the CPU, under Docker as the caller with no network, and writes the log to + `OUT.log`. +- `~/.config/rad-export/config.json` pins what auto cannot know: `image`, `out_dir`, + `recipes{model:recipe}`, `drafts{recipe:dir}`, `bases{repo:dir}`. +- It is Python 3 with no dependencies; merging needs `torch`/`safetensors`, fetching a missing base or + drafter needs `huggingface_hub`. The image carries no Python; extract the script with + `docker run --rm --entrypoint busybox stilldeadcode/radiance cat /opt/radiance/bin/rad-export`. + +## Scoring a container (the KL mode) + +```sh +# once, on the bf16 container +radiance --model bf16.rad --kld-record /srv/kld/q38 --kld-corpus corpus.jsonl +# each candidate, same flags otherwise +radiance --model quant.rad --kld-ref /srv/kld/q38 --kld-out report.json +``` + +- The corpus is JSONL, one `{"prompt", "score_from"?, "source"?}` a line; only `prompt` is required. + Positions `[score_from, n-1)` are scored; `source` groups the report. Keep the scoring corpus + disjoint from the calibration corpus. +- `--kld-record` writes the reference's log-probabilities (f16 over the vocabulary, the 16 most + likely tokens exactly, the token ids). DIR is never overwritten and is read only once `kld.json` + exists, which is written last. +- `--kld-ref` scores this model's f32 logits against stored rows: mean, median, 99/99.9/99.99th + percentile KL, top-1 agreement, and both models' perplexity. `--kld-out` writes the report; `FILE.rows` + is `[positions, 4]` f32 (KL, both NLLs, argmax agreement) for position-by-position comparison. +- The mode turns speculation and the prefix cache off, turns the step pipeline off, and refuses an + explicit spec depth above 0. The logits buffer holds a row per token of a step, so a large + `--max-num-batched-tokens` costs pinned host memory as well as VRAM. +- `--kld-lora NAME` scores every document through a served adapter, against a reference recorded + from the merged checkpoint; `--kld-lora-sources PREFIX` runs only matching documents through it, + the rest as the base, in the same steps — each document's rows should be byte-identical to the + all-adapter and all-base runs. + +The gate the fleet uses (mean KL ≤ 0.03, top-1 ≥ 95 %) is on [[frank-archon]] and +[[Quantization Production]]; the mode here is only the instrument. + +## Convert laws paid for in full + +- **`--plan-only` first.** A plan that resolves every op will boot; it costs seconds against an hour + of I/O on a large checkpoint. +- **Declare runs at the step and context bounds, and some graphs depend on them.** Qwen4-Exp sizes + its QSA buffers by `--max-ctx` and refuses a `--max-tok`/context pair whose score plane would pass + 2 GiB; the defaults (8192 tokens, the checkpoint's training context) fit most models, and a + generator's largest pass may override `--max-tok`. +- **Scale names follow the tensor.** `scale_name()` appends `_scale_inv` to the *tensor* name, so a + stacked `…experts.gate_up_proj` keeps its scales at `…gate_up_proj_scale_inv`. +- **File shadowing: first wins.** The reader opens index-named files sorted and emplaces every tensor + each file physically holds, first-wins. A merged PLE shard must sort first + (`00-ple.merged.safetensors`) or a raw shard wins the name. +- **Never write through a staging symlink.** A checkpoint may symlink its raw index; `open(path,"w")` + writes through it and corrupts the checkpoint. Write a temp file and `os.replace`. +- **One encoding per expert table.** A whole expert kept bf16 is the only thing legal beside a + table; a split tier (down=fp8, gate_up=w4) is unservable. +- **The descriptor pair has no fallback** (above), and **convert is card-less**: GPTQ solves on CPU + BLAS, so the GPU is irrelevant to production and the card only decides what can be decoded. + +## Related + +- [[Radiance Model Families]] — the (arch id, quant descriptor) table and the command per model. +- [[Radiance Plugins]] — quantiser and kernel-plugin contracts behind these options. +- [[Radiance Tools]] — `rad-info`, `rad-kbench`, `rad-tune`. +- [[Quantization Production]] — the fleet's calibration/KLD campaign and its gates. +- [[frank-archon]] — the deployment whose recipes and convert laws these are. diff --git a/serverAppConfigs-docs/LLM/Radiance Engine.md b/serverAppConfigs-docs/LLM/Radiance Engine.md new file mode 100644 index 0000000..75af0ac --- /dev/null +++ b/serverAppConfigs-docs/LLM/Radiance Engine.md @@ -0,0 +1,140 @@ +--- +type: "reference" +title: "Radiance Engine — the inference server behind frank-archon" +description: "The radiance inference engine: its shape, plugin model, container format, and how the reference pages fit together." +tags: + - llm + - radiance + - reference +timestamp: 2026-10-30 +--- + +# Radiance Engine — shape, plugin model, container + +Radiance is StillDeadcode's inference server for AMD GPUs, written in C++20 and HIP: language +models, plus image, video, speech and music generation and embeddings. One HTTP front end speaks +OpenAI's chat, completions, Responses, images, audio and video APIs, Anthropic's Messages API and +llama-server's native one. Nothing in the serving path is Python. + +The fleet's only radiance endpoint is [[frank-archon]] (2× R9700, TP=2, Dankserver). **This page +and its siblings own the engine reference; the live deployment shape, its Eras and its measured +laws stay on the endpoint page.** The numbers here describe upstream `main` as read 2026-10-30 +(release 1.14.0, plugin ABI 31). The fleet pin is older (1.8.1): a flag below that post-dates the pin +is a boot refusal, never a silent no-op, and the endpoint page records the CLI families that pin +itself retired relative to its own rollback target. + +Source: `https://codeberg.org/StillDeadcode/radiance` at `8da043c`. Engine docs live in `docs/` +(`GUIDE.md` for serving, `TOOLS.md` for tools, `PLUGIN.md` for plugins, `ARCHITECTURES.md` per +family, `OPS.md` for the op vocabulary, `spec.md` for the design). + +## The four moving parts + +```mermaid +graph LR + A[Checkpoint
safetensors / GGUF] -->|rad-convert + recipe| B[.rad container] + B --> C[radiance engine] + C --> D[OpenAI / Anthropic /
llama-server wire APIs] + P1[kernel library .so
libr4d, libavx, libref] --> C + P2[architecture .so
one per arch id × quant descriptor] --> C + P3[quantiser .so
libquant] -->|rad-convert| A +``` + +- **Engine** — `core/`: declare phase, placement/memory planner, scheduler, KV manager, sampler, + tokeniser, HTTP server, media preprocessing. +- **Kernel libraries** — implement ops for hardware. `libr4d` targets RDNA4 (`gfx1200`, + `gfx1201`); `libavx` serves the host backend (AVX-512/AVX2) with no GPU at all; `libref` is the + slow reference implementation every other library is checked against. +- **Architecture plugins** — declare one model family at one weight format: its weights, buffers, + KV groups and the ops it issues each step. The engine picks one by an **exact (architecture id, + quantisation descriptor) pair, with no fallback**. +- **Quantiser plugins** — `libquant`'s `rtn`, `gptq`, `awq`, `autoround`, `paroquant`, `m2xfp4`, + `ggml`, `cast`, `import`; they write a weight's numbers into an encoding's planes. +- **Tools** — `rad-info`, `rad-convert`, `rad-export`, `rad-kbench`, `rad-tune`, `rad-schemas`, + `rad-tokcheck`, plus `rad-chatparse`, `rad-avx-check`, `r4d_selftest` in the build tree. + +## Two phases, one static graph + +`rad_arch_declare` runs once per rank and names every weight, buffer, KV group and op, in order. +The core resolves every op against the kernel hierarchy per band, per domain, and completes the +whole plan before a byte of weights is read. `rad_arch_step` then only *issues* handles obtained at +declare; it may not allocate and may not synchronise. A model whose declare succeeds is fully known +statically, which is what makes `--debug-graph`, the buffer liveness plan and `rad-convert`'s +plan-only check possible. + +Consequences a plugin author must respect: + +- An op that never resolves is `RAD_NULL_HANDLE` and only fails if issued. A fusion is a *selection*: + ask for the fused op, emit the chain when `rad_op_resolved` is false. +- Declaration order is the buffer planner's liveness. A drafter must be declared after the trunk + it runs behind, or trunk and drafter buffers silently share bytes. +- Every op in the graph must have a row in a loaded kernel library. `libref` implements the + conventional vocabulary in `docs/OPS.md`; device-only ops (most fusions) exist only in `libr4d`. + +## The `.rad` container (RAD2) + +A container is one file holding everything the server needs: weights as **canonical encoding +planes**, the tokeniser (and optionally a second vocab), the chat template, metadata, the recipe it +was made with, and any draft head. It names no kernel library. + +- Planes are kernel-independent: codes, scales, zero points, codebooks — row-major, bit-packed + sub-byte where needed. A kernel rearranges them into its own layout at load (`layout`/`relayout` + hooks), losslessly, so a kernel library can change layouts without reconverting the model. +- Encodings are data, not an enum: `RadEncoding` names the scheme (`plain`, `affine`, or a + quantiser's own), the planes by role, the scale grouping, and a transform (`fwhtN`, `perm`, + `givens`). That is how one file can hold block-FP8, int4, MXFP4, NVFP4-imported, ParoQuant + rotations and llama.cpp grids. +- `rad-info` is the reader: sections (strings, metadata, directory, planes, encodings, vocab, + profile, data), `--meta`, `--vocab`, `--recipe`, `--plugins`, and `-v` for every tensor with its + encoding and what made it. + +See [[Radiance Conversion]] for producing one and [[Radiance Tools]] for inspecting it. + +## Selection key and version law + +- A checkpoint's architecture id is its `config.json` `model_type` without underscores; `rad-convert + --arch ID` overrides it. `rad_arch_quant` supplies the quantisation descriptor (empty for an + unquantised checkpoint). +- **No fallback.** A raw official FP8 checkpoint whose `quantization_config.quant_method` is `fp8` + is not the pair the `qwen4exp_fp8` plugin claims (`(qwen4exp, "")`), so both serve and convert + refuse it until the descriptor is changed with `--set` or a staged `config.json`. The deployment + lesson is on [[frank-archon]]; the mechanism is on [[Radiance Conversion]]. +- `RAD_ABI_VERSION` covers every header a plugin compiles against. A plugin reporting a different + number is refused at load, with both numbers printed. It moves on any shape change, not on a new + optional export; rebuild when it changes. Release version and ABI are independent: `/version` + reports the release, `rad-info --plugins` reports what loaded. + +## Environment + +| variable | meaning | +|---|---| +| `RADIANCE_HOME` | plugin search path: `:`-separated homes, each holding `kernels/`, `architectures/`, `quantizers/`; first wins, same-name files shadow later homes. Compiled-in default is `/share/radiance` | +| `RADIANCE_KERNELS` | default for `--kernels`: ordered kernel libraries, first wins; unnamed libraries are not loaded | +| `RADIANCE_TUNE` | pin a kernel's tuned axes for a measurement (a measurement aid, not a deployment setting) | +| `RADIANCE_CALIB_DIR` | a serve run records per-layer input Gram matrices here (the GPTQ calibration tap) | +| `RADIANCE_CALIB_TOKENS` | tokens after which the tap stops and writes (default 65536) | +| `RADIANCE_LOG`, `RADIANCE_LOG_STEPS` | log level; per-step token counts | +| `RADIANCE_DEBUG_RESID` | residual-stream probe per (layer, site) | +| `RADIANCE_DFLASH_Q_TEMP` | temperature of a DFlash2 coupled walk's candidate softmax | + +The full list of debug knobs (`RADIANCE_DEBUG_*`, `RADIANCE_DUMP_*`, `RADIANCE_OP_ORACLE`, …) is in +the source's `getenv` calls; none is a deployment setting. + +## Where the pages are + +| Page | Owns | +|---|---| +| [[Radiance Serve Options]] | every serve flag, grouped, with defaults and the law behind each knob | +| [[Radiance HTTP API]] | endpoints, request fields, streaming, tool calls, refusal rules | +| [[Radiance Conversion]] | `rad-convert`, the recipe language, quantisers, imports, reuse, `rad-export`, KL scoring | +| [[Radiance Model Families]] | which plugin claims which model, and the conversion command per family | +| [[Radiance Plugins]] | kernel and quantiser plugins: ABI, rows, hooks, encodings, build and checks | +| [[Radiance Architecture Plugins]] | declaring a model: weights, buffers, KV groups, ops, step, probe | +| [[Radiance Tools]] | `rad-info`, `rad-kbench`, `rad-tune`, `rad-schemas`, `rad-tokcheck`, harnesses | +| [[Radiance Operations]] | the container contract, host prerequisites, readiness, ledgers, restarts | +| [[frank-archon]] | the fleet's live radiance endpoint: shape, Eras, rollback, measured laws | + +## Sources + +- `README.md`, `docs/GUIDE.md` §1–§4, `docs/TOOLS.md`, `docs/PLUGIN.md`, `docs/DOCKER.md`, + `docs/ARCHITECTURES.md`, `docs/OPS.md`, `spec.md` §2–§4, `abi/*.h`, `core/config.cpp`, + `core/rad_core.h` at commit `8da043c` (codeberg.org/StillDeadcode/radiance). diff --git a/serverAppConfigs-docs/LLM/Radiance HTTP API.md b/serverAppConfigs-docs/LLM/Radiance HTTP API.md new file mode 100644 index 0000000..6f8f531 --- /dev/null +++ b/serverAppConfigs-docs/LLM/Radiance HTTP API.md @@ -0,0 +1,198 @@ +--- +type: "reference" +title: "Radiance HTTP API — endpoints, fields, refusals" +description: "The HTTP surface radiance serves: endpoints, request fields, streaming, tools, and refusal rules." +tags: + - llm + - radiance + - api + - reference +timestamp: 2026-10-30 +--- + +# Radiance HTTP API + +One server serves one model. It answers OpenAI's chat/completions/Responses/images/audio/video APIs, +Anthropic's Messages API and llama-server's native one, so their clients and SDKs work unchanged. +`model` in a request is accepted and **not enforced**. + +## Endpoints + +| route | what it serves | +|---|---| +| `POST /v1/chat/completions` | chat: tools, reasoning, streaming, grammars | +| `POST /v1/completions` | raw text completion | +| `POST /v1/embeddings` | an embedding head, else 501 | +| `POST /v1/images/generations`, `/v1/images/edits` | OpenAI images API; needs an image model, else 501 | +| `POST /v1/systemone` | LiquidAI d1 decisions: named yes/no, choice and score questions as probabilities; needs a d1 container, else 501 | +| `POST /v1/audio/transcriptions` | OpenAI speech-to-text; needs a transcribing plugin, else 501 | +| `GET /v1/realtime` (WebSocket) | streaming transcription session | +| `POST /v1/audio/speech` | speech (Breeze-TTS-2, drama-3) or a song (MiniMax-Music3), whole or streamed | +| `GET /v1/audio/speech/stream` (WebSocket) | text streamed in, each sentence's audio out | +| `POST /v1/videos`; `GET /v1/videos[/:id[/content]]`; `DELETE /v1/videos/:id` | video jobs made in the background, downloaded as MP4 | +| `POST /v1/messages`, `/v1/messages/count_tokens` | Anthropic Messages, for Claude Code and the Anthropic SDKs | +| `POST /v1/responses`, `/v1/responses/input_tokens` | OpenAI Responses, for Codex and SDK clients | +| `POST /completion`, `/completions`, `/infill` | llama-server native completion and fill-in-the-middle | +| `GET /v1/models`, `/v1/models/:id` | what is loaded, with `max_model_len`, `root`, `capabilities` | +| `GET /health`, `/v1/health`, `/ping`, `POST /ping` | liveness | +| `GET /load`, `/version`, `/props`, `/server_info`, `/diagnostics` | load; release version; llama-server props; full configuration; the startup report as JSON (`/diagnostics` is same-machine-only without an API key) | +| `POST /apply-template`, `/v1/chat/completions/input_tokens` | the rendered prompt; its token count | +| `GET /metrics` | Prometheus | +| `POST /tokenize`, `/detokenize`; `GET /get_tokenizer_info` | vLLM's shape and llama-server's at once | +| `POST /reset_prefix_cache`; `DELETE /sessions/