serverAppConfigs/serverAppConfigs-docs/LLM/Radiance Conversion.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
howto	Radiance Conversion — checkpoints into .rad containers	Turning a checkpoint into a .rad container: rad-convert, the recipe language, quantisers, reuse and export.	llm radiance quantization howto	2026-10-30

Radiance conversion

rad-convert turns a Hugging Face checkpoint into a .rad container, choosing each weight's encoding from a recipe. rad-export wraps it for finetunes (a merged checkpoint or a PEFT adapter) with the stock model's recipe. Neither opens a GPU: conversion is CPU-only and safe beside a running server.

Fetch checkpoints with scripts/hfget.sh <org/repo> <dir> (resumable; reads .safetensors, skips .bin/ONNX/GGUF when safetensors exist; HF_TOKEN for gated repos; writes a .hfget-manifest of content keys). radiance --model <owner/name> runs the same fetch itself.

What a conversion does

1. Read config.json (or GGUF metadata), and the shard index if present.
2. Load the architecture, kernel and quantiser plugins and run declare, as the engine does.
3. Resolve every op and ask each resolved kernel's layout hook about the planned encoding. A recipe no kernel can serve is refused here, before any I/O.
4. Write each logical weight once: the checkpoint's bytes where no rule matches, otherwise the planes the matching quantiser writes, a block of rows at a time.
5. Measure every quantised weight against its source through the core's decoder and print the worst.
6. Bake the tokeniser, chat template, metadata, recipe and any draft head.

build/bin/rad-convert /srv/models/hf/Qwen/Qwen3.8-27B -o qwen3.8-27b-bf16.rad \
    --recipe data/recipes/q38-27b-bf16.recipe
build/bin/rad-convert /srv/models/hf/Qwen/Qwen3.8-Flash-Next -o flashnext-bf16.rad --max-ctx 4096

Every rad-convert option

option	meaning
<input>	a .gguf, a .safetensors, an index.json, or a checkpoint directory
-o, --out FILE	the container to write
--recipe FILE	one rule a line: PATTERN QUANTISER key=value …; repeatable; first match decides
--quant RULE	PATTERN=QUANTISER[:key=value,…], placed ahead of the recipe's rules; repeatable
--list-quantizers	every loaded quantiser and the options it takes, then exit
--arch ID	override the architecture id read from the checkpoint
--home DIR	$RADIANCE_HOME (kernels, architectures, quantizers)
--kernels A:B	kernel libraries to load, first wins; declare checks the recipe against them
--tokenizer F	tokenizer.json; its declared chain is baked instead of reconstructed from a pre-tokenizer name. Required for a GGUF, without it the chain is a guess
--tokenizer2 F	a second tokenizer.json baked as the second vocab (a generator reading its prompt through two tokenisers); keep tokenizer_config.json beside it
--add FILE	merge one more .safetensors, tensor names kept; repeatable; a name in two files is refused
--config DIR	read config.json, generation_config.json, tokenizer.json from DIR for a single-file checkpoint
--chat-template F	a Jinja template baked in, overriding the checkpoint's (or its absence); trailing newlines stripped; without one /v1/chat/completions answers 501
--imatrix F	llama.cpp imatrix: expert-popularity profile, per-column importances, error weighting; required by ggml types iq2_xxs, iq2_xs, iq1_s
--set K=V	a metadata key, seen by declare and written into the container; repeatable
--max-tok N	the max_tok declare runs at (default 8192, or a generator's largest pass)
--max-ctx N	the context declare runs at (default: the checkpoint's training context)
--max-spec N	the max_spec declare runs at (default 4, or the block a drafter states); 0 leaves a draft head out
--draft-model DIR	a drafter shipped as its own repository, merged under draft.
--kv-cache-dtype T	bf16 or fp8: the cache width declare runs at; the container is identical at either
--reuse FILE	copy every weight an earlier container holds with the same encoding, quantiser, options and shape
--changed FILE	with --reuse: checkpoint tensors (one name a line) that differ; every weight reading one is converted
--in-place	with --reuse: extend that container; new weights go past its end, the header last; keeps <file>.pre-append
--lora FILE|DIR	merge a PEFT adapter into the checkpoint tensors before converting (W + strength · scale · B·A); repeatable, merged in order; each merged tensor joins --changed
--lora-strength F	strength for every --lora (default 1)
--lora-map FILE	JSON map for adapters whose module names differ: {"strip":[prefix…],"modules":{"<pat>":[target…]}}; one * matches a block index; several targets mean a fused module (rows split in order)
--plan-only	declare, plan every encoding, print the plan, stop before writing
--report N	print the N worst quantisation errors (default 20)
-v, --verbose	per-weight detail

--reuse matches a weight on its encoding, quantiser, options and shape — not on its source bytes — and carries whatever calibration made the old planes. Use it only when source and calibration are unchanged, or name the changed tensors with --changed. --in-place refuses to change a weight the container holds or to add an expert.

The recipe language

# PATTERN                      QUANTISER  OPTIONS
blk.*.ffn_*_exps.*.weight      gptq       codes=i4 group=128 scale=bf16 transform=fwht128 calib=$CALIB
blk.*.ple_ngram.weight         rtn        codes=fp8_e4m3 block=*x* scale=bf16 rule=fixed scale_value=1.99317932128906e-4
output.weight                  rtn        codes=fp8_e4m3 block=128x128 scale=bf16
blk.*.weight                   rtn        codes=fp8_e4m3 block=128x128 scale=bf16

• One rule a line: PATTERN QUANTISER [key=value …]. # starts a comment.
• PATTERN is a glob over the whole logical weight name: * crosses dots, ? is one character.
• The first matching rule decides. No rule → the checkpoint's own encoding, which must be one the engine serves as is: bf16, f16, f32, or block-scaled FP8 (X.weight with X.weight_scale_inv). Anything else is named and refused.
• $NAME / ${NAME} in a value comes from the environment; an unset variable is refused, never expanded empty (so an empty $CALIB cannot silently produce an uncalibrated convert).
• A quantiser may decline a weight (RAD_E_UNSUPPORTED), which keeps the checkpoint's encoding — rtn declines a rank-1 tensor, so a pattern that also matches a norm leaves the norm alone.
• An option the quantiser does not declare is refused, by name, before anything is quantised.
• Two plugins naming the same quantiser are refused, both named.

Quantisers

quantiser	what it does
rtn	round to nearest on a grid; each block's scale chosen by rule
gptq	error feedback against the layer's input Hessian (calib=DIR, damped by damp = 0.01 of the mean diagonal by default); act_order=1 quantises columns by descending input energy and stores the permutation; cd=N adds coordinate-descent sweeps
awq	per-input-column scaling to a searched power, then a per-block clip search, both against the Hessian; column scales are the last scale level
autoround	sign-gradient descent on a rounding offset per value and a range factor per block; iters steps; slow (a 27B dense model takes most of a working day at defaults; iters=50 a quarter)
paroquant	channel scaling plus stages Givens rotations learned against the Hessian, then AutoRound on the rotated weight; the givens transform
m2xfp4	M2XFP's weight-side Sg-EM: MXFP4 plus one metadata byte per group of 32; its own m2xfp4 scheme; adaptive=0 pins the bias, wide=1 widens it
ggml	a llama.cpp format (q4_0, q4_1, q5_0, q5_1, q8_0, q2_K…q6_K, iq4_nl, iq4_xs, iq2_xxs, iq2_xs, iq2_s, iq3_xxs, iq3_s, iq1_s, iq1_m), by llama.cpp's own search, imatrix-weighted, codes byte for byte
cast	a float weight in another float dtype (dtype= f32, bf16, f16, fp8_e4m3, fp8_e5m2), no scale
import	rad-convert's own, not a plugin: write the numbers a quantised release ships, in the spelling the kernels read, instead of quantising again

Every quantiser except m2xfp4, ggml, cast and import takes the grid options:

option	values
codes	i2…i8, i16, i32 symmetric; u1…u8, u16 with zero=; fp8_e4m3, fp8_e5m2, fp6_e2m3, fp6_e3m2, fp4_e2m1
table	a codebook the codes index: nf4, iq4nl, w4nl, w5nl, binary, ternary, or ;-separated values
group=N / block=RxC	where the scales sit; * is a whole extent
scale	f32, bf16, f16, fp8_e4m3, e8m0; under scale2 also an integer sub-scale u4…u8, i6, i8
scale2, group2 / block2	a second scale level the first is relative to (default the whole tensor)
scale2_value	the second level fixed at this value, exact in scale2's dtype
zero	an integer zero point per block — an asymmetric grid
transform=fwhtN	rotate each row by the Hadamard first
rule	absmax, mx (OCP shared exponent), fixed (scale_value=), search (least squared error over candidates and refits, imatrix-weighted)
clamp	a signed code's range: full or sym

Named grids: MXFP4 = codes=fp4_e2m1 group=32 scale=e8m0; NVFP4 = codes=fp4_e2m1 group=16 scale=fp8_e4m3 scale2=f32; NF4 = table=nf4 group=64; a K-quant-like grid = codes=i4 group=32 scale=u6 scale2=f16 group2=256; a ternary BitNet weight = table=ternary.

A container can hold only what some kernel reads. rad-convert checks every planned encoding against the layout hooks of the kernels declare resolved and refuses one no kernel accepts. libr4d reads the plain floats; block-scaled fp8 and fp8 rows; int8 block rows plain or rotated; routed experts as fp8, int4 (i4*bf16[1x128]/fwht128) or the w4nl/w5nl codebooks under E4M3 scales; the dense w4, w8 and mxfp4 GEMM rows; and the 2-bit draft head. See Radiance Plugins for the encoding model.

Importing a release someone else quantised

These forms are read weight by weight and written in the spelling the kernels read; a recipe rule re-quantises from the exact dequantisation where the family's kernels want another form.

quantization_config	read as	notes
quark MXFP4 (fp4, group 32, e8m0)	fp4_e2m1*e8m0[1x32]	byte for byte; also servable as it stands
paroquant_mxfp4	fp4_e2m1*e8m0[1x32]/givens	codes/exponents byte for byte plus learned rotations as givens tables
paroquant, int5-bitplane, plain paroquant	u5*f16[1x128]-u8[1x128]/givens, u4*…	AWQ-packed nibbles re-laid as a code bit stream; f16 scales, integer zeros, rotations
compressed-tensors pack-quantized, 4-bit int, group, act-order none/static/weight	i4*<scale>[1xG]	weight_packed made signed beside weight_scale; group/dynamic reorder is refused
same with symmetric:false (AWQ)	u4*<scale>[1xG]-u4[1xG]	nibbles plus weight_zero_point, bit for bit
auto-round auto_round:auto_gptq, gptq v1	i4*bf16[1x128] or i8*bf16[1x128]	scales rounded to bf16; convert-time; import keeps shipped f16
nvfp4-pack-quantized	fp4_e2m1*f32[1x16]	weight_scale / weight_global_scale; no kernel serves NVFP4, so it always takes a rule
modelopt NVFP4 / FP8 / MIXED_PRECISION	NVFP4 fp4_e2m1*f32[1x16]; FP8 fp8_e4m3*f32[1x*]	ModelOpt's second scale multiplies; input_scale is not read
compressed-tensors all-FP8 groups	fp8_e4m3*<scale>[1x*] or [RxC]	nothing imported: stock FP8 with its scale; a per-tensor scale is written once a row
GGUF Q8_0, Q4_K, Q6_K	i8*f16[1x32], affine u4/u6…, i6*i8…	blocks split bit for bit; not servable as they stand

The width is read per tensor, so a mixed release imports int4 experts and int8 projections together. What an import cannot read (an asymmetric GPTQ zero point, a group-reordering act-order index, a GPTQ group under 128, a missing/zero global scale, an unlisted format) is refused by name. The release's quantization_config names the tool, not a served form, so it is kept as imported_quantization_config.*.

import options: codes=uN makes a symmetric grid unsigned under a zero point of 2^(N-1); zero= the zero plane's dtype (u4, u8); scale= f16/bf16/f32, each shipped scale rounded to nearest; block=RxC repeats an FP8 scale shipped per row/tensor over that grid. It reads an imported integer grid or FP8 with a per-row/per-tensor scale and keeps a plain float weight as it is.

NVFP4 to MXFP4 rounds twice (e4m3 per 16 → power of two per 32), so it loses more than MXFP4 made from bf16. Prefer a calibrated MXFP4 container when one exists; the quality verdict is a KL gate, not a file size.

The descriptor law (--set)

The architecture plugin is chosen by the exact (arch id, quant descriptor) pair. A convert-time choice that must also hold at serve time is stated once with --set, which hands it to declare and writes it into the container. Storage format is not such a choice (the encoding is in the container and the plugin reads it there), but the serving plugin is: a quantization_config.* key re-derives the descriptor.

rad-convert <bf16 Qwen3.8-27B> -o q38-27b-fp8.rad \
    --recipe data/recipes/q38-27b-fp8block-df2.recipe \
    --set quantization_config.quant_method=fp8 --set quantization_config.fmt=e4m3 \
    --set "quantization_config.weight_block_size=128 128" \
    --set quantization_config.activation_scheme=dynamic

A bf16 checkpoint whose recipe writes Qwen's block FP8 must therefore be declared as that format; otherwise the selection pair does not match any plugin and the convert (or the boot) is refused. The same law bites the other way when a release's own quant_method names a pair no plugin claims.

Inspect and validate the result

rad-info out.rad                  # architecture, source quant, recipe, sections
rad-info --recipe out.rad         # the expanded rules — did the intended one win?
rad-info -v out.rad | head -40    # every tensor, its encoding, and what made it
rad-kbench -m out.rad --home $RADIANCE_HOME --report kbench.md
radiance --model out.rad --debug-graph     # every op resolves

The converter's own error report (the worst --report N weights) measures each quantised weight against its source through the core decoder. rad-kbench adds the artifact test: weight operands read from the container and passed through each kernel's layout/relayout hooks. Both are on Radiance Tools.

Calibration (GPTQ, AWQ, AutoRound, ParoQuant)

calib=DIR is a directory an engine run wrote with RADIANCE_CALIB_DIR set: each layer's Gram matrices of its input, in the domain the kernel multiplies it in. A block that characterises an activation declares an f32 [K,K] PERSIST buffer and a gram_accum op; at RADIANCE_CALIB_TOKENS (default 65536) the run writes the files.

Three-step bootstrap: convert with plain rtn → serve that container with RADIANCE_CALIB_DIR → push real prompts through prefill → convert again with the Hessians. The second moment is robust to the first container's small quantisation error. A weight with no file is quantised uncalibrated and the converter says so; awq, autoround and paroquant fall back to the imatrix diagonal when one is given.

• The accumulator must be PERSIST: a transient buffer may share bytes and be overwritten between the launches it sums.
• Files are per rank; the consumer sums them. Under expert parallelism a rank's down input covers only its experts' tokens; gate_up input is the full token set on every rank, so the sum scales H by world size — GPTQ is invariant to a positive scale.
• Under tensor parallelism a column-split linear sees only its rank's columns, so a tap may write gram.r<rank>.<stem>.part.bin; libquant places the parts on the diagonal in rank order, leaving cross-rank products zero. They must cover K and be in the model's own domain.
• --imatrix supplies the expert-popularity profile the placement planner ranks units by, the per-column importances a quantiser uses, and the error report's weighting.

Reuse, drafters, merging

# re-quantise only what changed
rad-convert $SRC -o new.rad --recipe new.recipe --reuse old.rad --changed changed.txt

# merge a drafter shipped as its own repo (DFlash2, DSpark, MTP assistant)
rad-convert $SRC -o out.rad --draft-model /srv/models/z-lab/Qwen3.8-27B-DFlash2 \
    --recipe data/recipes/q38-27b-mxfp4-df2.recipe

# merge a PEFT adapter at convert time
rad-convert $SRC -o tuned.rad --lora ~/run/adapter --lora-strength 1.0 \
    --recipe data/recipes/q38-27b-mxfp4-df2.recipe

• A drafter's tensors go in under draft., its config under draft. keys. It has no embedding and no lm_head of its own; it uses the target's.
• --lora-map exists for adapters trained on names the checkpoint does not use; a fused target (qkv) takes several targets and splits the adapter's rows in order.
• --add + --config + --tokenizer2 assemble a ComfyUI-style release of separate single files (Anima: diffusion model, text encoder, VAE). A name in two files is refused.

rad-export — finetunes without hand-writing the command

rad-export ~/outputs/my-run/checkpoint-400 --dry-run   # what it would run
rad-export ~/outputs/my-run/checkpoint-400             # adapter -> <name>.rad
rad-export ~/exports/merged -o ~/models/mine.rad --recipe q38-27b-fp8block-df2

flag	default	meaning
SRC	required	a merged 16-bit checkpoint (config.json + *.safetensors) or a PEFT adapter (adapter_config.json + adapter_model.safetensors)
-o, --out FILE	SRC's folder name + .rad, beside SRC or in out_dir	the container
--recipe NAME|FILE|auto	auto	a recipe name, a file, or the one recipe whose header names the base model, takes a bf16 checkpoint and needs no calibration
--base DIR	the adapter's base_model_name_or_path, from disk or the HF cache, else downloaded	the base a LoRA merges into
--draft DIR|ORG/NAME	the config, else the Hub repo the recipe's usage line names	the drafter when the recipe merges one; a usage line with a placeholder asks for --draft
--image IMAGE	$RADIANCE_IMAGE, the config's, then stilldeadcode/radiance	run rad-convert in this image (docker, CPU-only, no network, not pulled)
--rad-convert PATH	—	run this binary instead of docker; recipes come from its share/radiance
--memory M	—	docker memory limit for the convert
--dry-run	—	resolve recipe, drafter and paths; print the rad-convert line; write nothing
--keep-merged	—	keep the merged checkpoint beside OUT
--config FILE	~/.config/rad-export/config.json	the config below
--list-recipes [--json]	—	list the recipes the image/installed tool carries

• Input: a merged 16-bit checkpoint (config.json + *.safetensors) or a PEFT LoRA adapter (adapter_config.json + adapter_model.safetensors).
• An adapter is merged exactly as PEFT's merge_and_unload computes it on CPU: scale = alpha/r (alpha/√r with rsLoRA), rank_pattern/alpha_pattern honoured, delta in fp32 rounded to the adapter's dtype. It streams tensor by tensor into 1 GiB shards, so a 27B merge needs a few GiB of RAM, not the model twice. Plain linear LoRA only: embedding LoRA, trained biases, DoRA, saved modules, convolution LoRA, a rank other than the config's, and a missing target are refused before anything is written. The adapter's tokeniser/template replace the base's where shipped.
• The recipe comes from the image (/opt/radiance/share/radiance/recipes) or the installed rad-convert's share; auto matches the base model's name in recipe headers and asks when several fit. A recipe needing calibration is listed but never picked by auto — a finetune's Hessians are not the stock model's.
• It runs rad-convert on the CPU, under Docker as the caller with no network, and writes the log to OUT.log.
• ~/.config/rad-export/config.json pins what auto cannot know: image, out_dir, recipes{model:recipe}, drafts{recipe:dir}, bases{repo:dir}.
• It is Python 3 with no dependencies; merging needs torch/safetensors, fetching a missing base or drafter needs huggingface_hub. The image carries no Python; extract the script with docker run --rm --entrypoint busybox stilldeadcode/radiance cat /opt/radiance/bin/rad-export.

Scoring a container (the KL mode)

# once, on the bf16 container
radiance --model bf16.rad --kld-record /srv/kld/q38 --kld-corpus corpus.jsonl
# each candidate, same flags otherwise
radiance --model quant.rad --kld-ref /srv/kld/q38 --kld-out report.json

• The corpus is JSONL, one {"prompt", "score_from"?, "source"?} a line; only prompt is required. Positions [score_from, n-1) are scored; source groups the report. Keep the scoring corpus disjoint from the calibration corpus.
• --kld-record writes the reference's log-probabilities (f16 over the vocabulary, the 16 most likely tokens exactly, the token ids). DIR is never overwritten and is read only once kld.json exists, which is written last.
• --kld-ref scores this model's f32 logits against stored rows: mean, median, 99/99.9/99.99th percentile KL, top-1 agreement, and both models' perplexity. --kld-out writes the report; FILE.rows is [positions, 4] f32 (KL, both NLLs, argmax agreement) for position-by-position comparison.
• The mode turns speculation and the prefix cache off, turns the step pipeline off, and refuses an explicit spec depth above 0. The logits buffer holds a row per token of a step, so a large --max-num-batched-tokens costs pinned host memory as well as VRAM.
• --kld-lora NAME scores every document through a served adapter, against a reference recorded from the merged checkpoint; --kld-lora-sources PREFIX runs only matching documents through it, the rest as the base, in the same steps — each document's rows should be byte-identical to the all-adapter and all-base runs.

The gate the fleet uses (mean KL ≤ 0.03, top-1 ≥ 95 %) is on frank-archon and Quantization Production; the mode here is only the instrument.

Convert laws paid for in full

• --plan-only first. A plan that resolves every op will boot; it costs seconds against an hour of I/O on a large checkpoint.
• Declare runs at the step and context bounds, and some graphs depend on them. Qwen4-Exp sizes its QSA buffers by --max-ctx and refuses a --max-tok/context pair whose score plane would pass 2 GiB; the defaults (8192 tokens, the checkpoint's training context) fit most models, and a generator's largest pass may override --max-tok.
• Scale names follow the tensor. scale_name() appends _scale_inv to the tensor name, so a stacked …experts.gate_up_proj keeps its scales at …gate_up_proj_scale_inv.
• File shadowing: first wins. The reader opens index-named files sorted and emplaces every tensor each file physically holds, first-wins. A merged PLE shard must sort first (00-ple.merged.safetensors) or a raw shard wins the name.
• Never write through a staging symlink. A checkpoint may symlink its raw index; open(path,"w") writes through it and corrupts the checkpoint. Write a temp file and os.replace.
• One encoding per expert table. A whole expert kept bf16 is the only thing legal beside a table; a split tier (down=fp8, gate_up=w4) is unservable.
• The descriptor pair has no fallback (above), and convert is card-less: GPTQ solves on CPU BLAS, so the GPU is irrelevant to production and the card only decides what can be decoded.

Related

• Radiance Model Families — the (arch id, quant descriptor) table and the command per model.
• Radiance Plugins — quantiser and kernel-plugin contracts behind these options.
• Radiance Tools — rad-info, rad-kbench, rad-tune.
• Quantization Production — the fleet's calibration/KLD campaign and its gates.
• frank-archon — the deployment whose recipes and convert laws these are.