docs(radiance): reference the engine's options, plugins and conversion
Nine pages under LLM/ document the upstream radiance engine at 8da043c: - engine shape, plugin model, RAD2 container, ABI and environment - every serve flag grouped with defaults and the law behind each knob - the HTTP surface, request fields and refusal classes - rad-convert, the recipe language, quantisers, imports, reuse, rad-export, KL scoring - the (arch id, quant) family table and the recipe of record per model - kernel/quantiser plugin authoring, encodings, build and checks - architecture plugin authoring: declare, step, probe, lane and media rules - rad-info/radbench/radtune and the shell harnesses - the container contract, host prerequisites, readiness and restart triage Also completes the dangling Related line on the frank-archon endpoint page.
This commit is contained in:
1 parent
da5eb968f9
commit
3a04e34414
12 files changed
+2319
-2
No files matched your search
@@ -0,0 +1,354 @@
|
||||
---
|
||||
type: "howto"
|
||||
title: "Radiance Conversion — checkpoints into .rad containers"
|
||||
description: "Turning a checkpoint into a .rad container: rad-convert, the recipe language, quantisers, reuse and export."
|
||||
tags:
|
||||
- llm
|
||||
- radiance
|
||||
- quantization
|
||||
- howto
|
||||
timestamp: 2026-10-30
|
||||
---
|
||||
|
||||
# Radiance conversion
|
||||
|
||||
`rad-convert` turns a Hugging Face checkpoint into a `.rad` container, choosing each weight's
|
||||
encoding from a **recipe**. `rad-export` wraps it for finetunes (a merged checkpoint or a PEFT
|
||||
adapter) with the stock model's recipe. Neither opens a GPU: conversion is CPU-only and safe beside a
|
||||
running server.
|
||||
|
||||
Fetch checkpoints with `scripts/hfget.sh <org/repo> <dir>` (resumable; reads `.safetensors`,
|
||||
skips `.bin`/ONNX/GGUF when safetensors exist; `HF_TOKEN` for gated repos; writes a
|
||||
`.hfget-manifest` of content keys). `radiance --model <owner/name>` runs the same fetch itself.
|
||||
|
||||
## What a conversion does
|
||||
|
||||
1. Read `config.json` (or GGUF metadata), and the shard index if present.
|
||||
2. Load the architecture, kernel and quantiser plugins and run **declare**, as the engine does.
|
||||
3. Resolve every op and ask each resolved kernel's `layout` hook about the planned encoding. A
|
||||
recipe no kernel can serve is refused **here**, before any I/O.
|
||||
4. Write each logical weight once: the checkpoint's bytes where no rule matches, otherwise the
|
||||
planes the matching quantiser writes, a block of rows at a time.
|
||||
5. Measure every quantised weight against its source through the core's decoder and print the worst.
|
||||
6. Bake the tokeniser, chat template, metadata, recipe and any draft head.
|
||||
|
||||
```sh
|
||||
build/bin/rad-convert /srv/models/hf/Qwen/Qwen3.8-27B -o qwen3.8-27b-bf16.rad \
|
||||
--recipe data/recipes/q38-27b-bf16.recipe
|
||||
build/bin/rad-convert /srv/models/hf/Qwen/Qwen3.8-Flash-Next -o flashnext-bf16.rad --max-ctx 4096
|
||||
```
|
||||
|
||||
## Every `rad-convert` option
|
||||
|
||||
| option | meaning |
|
||||
|---|---|
|
||||
| `<input>` | a `.gguf`, a `.safetensors`, an `index.json`, or a checkpoint directory |
|
||||
| `-o, --out FILE` | the container to write |
|
||||
| `--recipe FILE` | one rule a line: `PATTERN QUANTISER key=value …`; repeatable; first match decides |
|
||||
| `--quant RULE` | `PATTERN=QUANTISER[:key=value,…]`, placed **ahead of** the recipe's rules; repeatable |
|
||||
| `--list-quantizers` | every loaded quantiser and the options it takes, then exit |
|
||||
| `--arch ID` | override the architecture id read from the checkpoint |
|
||||
| `--home DIR` | `$RADIANCE_HOME` (kernels, architectures, quantizers) |
|
||||
| `--kernels A:B` | kernel libraries to load, first wins; declare checks the recipe against them |
|
||||
| `--tokenizer F` | `tokenizer.json`; its declared chain is baked instead of reconstructed from a pre-tokenizer name. Required for a GGUF, without it the chain is a guess |
|
||||
| `--tokenizer2 F` | a second `tokenizer.json` baked as the second vocab (a generator reading its prompt through two tokenisers); keep `tokenizer_config.json` beside it |
|
||||
| `--add FILE` | merge one more `.safetensors`, tensor names kept; repeatable; a name in two files is refused |
|
||||
| `--config DIR` | read `config.json`, `generation_config.json`, `tokenizer.json` from DIR for a single-file checkpoint |
|
||||
| `--chat-template F` | a Jinja template baked in, overriding the checkpoint's (or its absence); trailing newlines stripped; without one `/v1/chat/completions` answers 501 |
|
||||
| `--imatrix F` | llama.cpp imatrix: expert-popularity profile, per-column importances, error weighting; required by `ggml` types `iq2_xxs`, `iq2_xs`, `iq1_s` |
|
||||
| `--set K=V` | a metadata key, seen by declare **and** written into the container; repeatable |
|
||||
| `--max-tok N` | the `max_tok` declare runs at (default 8192, or a generator's largest pass) |
|
||||
| `--max-ctx N` | the context declare runs at (default: the checkpoint's training context) |
|
||||
| `--max-spec N` | the `max_spec` declare runs at (default 4, or the block a drafter states); `0` leaves a draft head out |
|
||||
| `--draft-model DIR` | a drafter shipped as its own repository, merged under `draft.` |
|
||||
| `--kv-cache-dtype T` | `bf16` or `fp8`: the cache width declare runs at; the container is identical at either |
|
||||
| `--reuse FILE` | copy every weight an earlier container holds with the same encoding, quantiser, options and shape |
|
||||
| `--changed FILE` | with `--reuse`: checkpoint tensors (one name a line) that differ; every weight reading one is converted |
|
||||
| `--in-place` | with `--reuse`: extend that container; new weights go past its end, the header last; keeps `<file>.pre-append` |
|
||||
| `--lora FILE\|DIR` | merge a PEFT adapter into the checkpoint tensors before converting (`W + strength · scale · B·A`); repeatable, merged in order; each merged tensor joins `--changed` |
|
||||
| `--lora-strength F` | strength for every `--lora` (default 1) |
|
||||
| `--lora-map FILE` | JSON map for adapters whose module names differ: `{"strip":[prefix…],"modules":{"<pat>":[target…]}}`; one `*` matches a block index; several targets mean a fused module (rows split in order) |
|
||||
| `--plan-only` | declare, plan every encoding, print the plan, stop before writing |
|
||||
| `--report N` | print the N worst quantisation errors (default 20) |
|
||||
| `-v, --verbose` | per-weight detail |
|
||||
|
||||
`--reuse` matches a weight on its **encoding, quantiser, options and shape — not on its source
|
||||
bytes** — and carries whatever calibration made the old planes. Use it only when source and
|
||||
calibration are unchanged, or name the changed tensors with `--changed`. `--in-place` refuses to
|
||||
change a weight the container holds or to add an expert.
|
||||
|
||||
## The recipe language
|
||||
|
||||
```
|
||||
# PATTERN QUANTISER OPTIONS
|
||||
blk.*.ffn_*_exps.*.weight gptq codes=i4 group=128 scale=bf16 transform=fwht128 calib=$CALIB
|
||||
blk.*.ple_ngram.weight rtn codes=fp8_e4m3 block=*x* scale=bf16 rule=fixed scale_value=1.99317932128906e-4
|
||||
output.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16
|
||||
blk.*.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16
|
||||
```
|
||||
|
||||
- One rule a line: `PATTERN QUANTISER [key=value …]`. `#` starts a comment.
|
||||
- `PATTERN` is a glob over the **whole logical weight name**: `*` crosses dots, `?` is one character.
|
||||
- **The first matching rule decides.** No rule → the checkpoint's own encoding, which must be one
|
||||
the engine serves as is: `bf16`, `f16`, `f32`, or block-scaled FP8 (`X.weight` with
|
||||
`X.weight_scale_inv`). Anything else is named and refused.
|
||||
- `$NAME` / `${NAME}` in a value comes from the environment; an **unset variable is refused**, never
|
||||
expanded empty (so an empty `$CALIB` cannot silently produce an uncalibrated convert).
|
||||
- A quantiser may **decline** a weight (`RAD_E_UNSUPPORTED`), which keeps the checkpoint's encoding
|
||||
— `rtn` declines a rank-1 tensor, so a pattern that also matches a norm leaves the norm alone.
|
||||
- An option the quantiser does not declare is refused, by name, before anything is quantised.
|
||||
- Two plugins naming the same quantiser are refused, both named.
|
||||
|
||||
## Quantisers
|
||||
|
||||
| quantiser | what it does |
|
||||
|---|---|
|
||||
| `rtn` | round to nearest on a grid; each block's scale chosen by `rule` |
|
||||
| `gptq` | error feedback against the layer's input Hessian (`calib=DIR`, damped by `damp` = 0.01 of the mean diagonal by default); `act_order=1` quantises columns by descending input energy and stores the permutation; `cd=N` adds coordinate-descent sweeps |
|
||||
| `awq` | per-input-column scaling to a searched power, then a per-block clip search, both against the Hessian; column scales are the last scale level |
|
||||
| `autoround` | sign-gradient descent on a rounding offset per value and a range factor per block; `iters` steps; slow (a 27B dense model takes most of a working day at defaults; `iters=50` a quarter) |
|
||||
| `paroquant` | channel scaling plus `stages` Givens rotations learned against the Hessian, then AutoRound on the rotated weight; the `givens` transform |
|
||||
| `m2xfp4` | M2XFP's weight-side Sg-EM: MXFP4 plus one metadata byte per group of 32; its own `m2xfp4` scheme; `adaptive=0` pins the bias, `wide=1` widens it |
|
||||
| `ggml` | a llama.cpp format (`q4_0`, `q4_1`, `q5_0`, `q5_1`, `q8_0`, `q2_K`…`q6_K`, `iq4_nl`, `iq4_xs`, `iq2_xxs`, `iq2_xs`, `iq2_s`, `iq3_xxs`, `iq3_s`, `iq1_s`, `iq1_m`), by llama.cpp's own search, imatrix-weighted, codes byte for byte |
|
||||
| `cast` | a float weight in another float dtype (`dtype=` f32, bf16, f16, fp8_e4m3, fp8_e5m2), no scale |
|
||||
| `import` | rad-convert's own, not a plugin: write the numbers a quantised release ships, in the spelling the kernels read, instead of quantising again |
|
||||
|
||||
Every quantiser except `m2xfp4`, `ggml`, `cast` and `import` takes the grid options:
|
||||
|
||||
| option | values |
|
||||
|---|---|
|
||||
| `codes` | `i2`…`i8`, `i16`, `i32` symmetric; `u1`…`u8`, `u16` with `zero=`; `fp8_e4m3`, `fp8_e5m2`, `fp6_e2m3`, `fp6_e3m2`, `fp4_e2m1` |
|
||||
| `table` | a codebook the codes index: `nf4`, `iq4nl`, `w4nl`, `w5nl`, `binary`, `ternary`, or `;`-separated values |
|
||||
| `group=N` / `block=RxC` | where the scales sit; `*` is a whole extent |
|
||||
| `scale` | f32, bf16, f16, fp8_e4m3, e8m0; under `scale2` also an integer sub-scale `u4`…`u8`, `i6`, `i8` |
|
||||
| `scale2`, `group2` / `block2` | a second scale level the first is relative to (default the whole tensor) |
|
||||
| `scale2_value` | the second level fixed at this value, exact in `scale2`'s dtype |
|
||||
| `zero` | an integer zero point per block — an asymmetric grid |
|
||||
| `transform=fwhtN` | rotate each row by the Hadamard first |
|
||||
| `rule` | `absmax`, `mx` (OCP shared exponent), `fixed` (`scale_value=`), `search` (least squared error over candidates and refits, imatrix-weighted) |
|
||||
| `clamp` | a signed code's range: `full` or `sym` |
|
||||
|
||||
Named grids: MXFP4 = `codes=fp4_e2m1 group=32 scale=e8m0`; NVFP4 =
|
||||
`codes=fp4_e2m1 group=16 scale=fp8_e4m3 scale2=f32`; NF4 = `table=nf4 group=64`; a K-quant-like grid
|
||||
= `codes=i4 group=32 scale=u6 scale2=f16 group2=256`; a ternary BitNet weight = `table=ternary`.
|
||||
|
||||
**A container can hold only what some kernel reads.** rad-convert checks every planned encoding
|
||||
against the `layout` hooks of the kernels declare resolved and refuses one no kernel accepts. libr4d
|
||||
reads the plain floats; block-scaled fp8 and fp8 rows; int8 block rows plain or rotated; routed
|
||||
experts as fp8, int4 (`i4*bf16[1x128]/fwht128`) or the `w4nl`/`w5nl` codebooks under E4M3 scales;
|
||||
the dense w4, w8 and mxfp4 GEMM rows; and the 2-bit draft head. See [[Radiance Plugins]] for the
|
||||
encoding model.
|
||||
|
||||
## Importing a release someone else quantised
|
||||
|
||||
These forms are read weight by weight and written in the spelling the kernels read; a recipe rule
|
||||
re-quantises from the exact dequantisation where the family's kernels want another form.
|
||||
|
||||
| `quantization_config` | read as | notes |
|
||||
|---|---|---|
|
||||
| `quark` MXFP4 (fp4, group 32, e8m0) | `fp4_e2m1*e8m0[1x32]` | byte for byte; also servable as it stands |
|
||||
| `paroquant_mxfp4` | `fp4_e2m1*e8m0[1x32]/givens` | codes/exponents byte for byte plus learned rotations as `givens` tables |
|
||||
| `paroquant`, `int5-bitplane`, plain `paroquant` | `u5*f16[1x128]-u8[1x128]/givens`, `u4*…` | AWQ-packed nibbles re-laid as a code bit stream; f16 scales, integer zeros, rotations |
|
||||
| `compressed-tensors` `pack-quantized`, 4-bit `int`, group, act-order none/static/weight | `i4*<scale>[1xG]` | `weight_packed` made signed beside `weight_scale`; group/dynamic reorder is refused |
|
||||
| same with `symmetric:false` (AWQ) | `u4*<scale>[1xG]-u4[1xG]` | nibbles plus `weight_zero_point`, bit for bit |
|
||||
| `auto-round` `auto_round:auto_gptq`, `gptq` v1 | `i4*bf16[1x128]` or `i8*bf16[1x128]` | scales rounded to bf16; convert-time; `import` keeps shipped f16 |
|
||||
| `nvfp4-pack-quantized` | `fp4_e2m1*f32[1x16]` | `weight_scale / weight_global_scale`; **no kernel serves NVFP4**, so it always takes a rule |
|
||||
| `modelopt` NVFP4 / FP8 / MIXED_PRECISION | NVFP4 `fp4_e2m1*f32[1x16]`; FP8 `fp8_e4m3*f32[1x*]` | ModelOpt's second scale **multiplies**; `input_scale` is not read |
|
||||
| compressed-tensors all-FP8 groups | `fp8_e4m3*<scale>[1x*]` or `[RxC]` | nothing imported: stock FP8 with its scale; a per-tensor scale is written once a row |
|
||||
| GGUF `Q8_0`, `Q4_K`, `Q6_K` | `i8*f16[1x32]`, affine u4/u6…, `i6*i8…` | blocks split bit for bit; **not servable as they stand** |
|
||||
|
||||
The width is read **per tensor**, so a mixed release imports int4 experts and int8 projections
|
||||
together. What an import cannot read (an asymmetric GPTQ zero point, a group-reordering act-order
|
||||
index, a GPTQ group under 128, a missing/zero global scale, an unlisted format) is refused by name.
|
||||
The release's `quantization_config` names the tool, not a served form, so it is kept as
|
||||
`imported_quantization_config.*`.
|
||||
|
||||
`import` options: `codes=uN` makes a symmetric grid unsigned under a zero point of `2^(N-1)`;
|
||||
`zero=` the zero plane's dtype (`u4`, `u8`); `scale=` f16/bf16/f32, each shipped scale rounded to
|
||||
nearest; `block=RxC` repeats an FP8 scale shipped per row/tensor over that grid. It reads an imported
|
||||
integer grid or FP8 with a per-row/per-tensor scale and keeps a plain float weight as it is.
|
||||
|
||||
**NVFP4 to MXFP4 rounds twice** (e4m3 per 16 → power of two per 32), so it loses more than MXFP4 made
|
||||
from bf16. Prefer a calibrated MXFP4 container when one exists; the quality verdict is a KL gate, not
|
||||
a file size.
|
||||
|
||||
## The descriptor law (`--set`)
|
||||
|
||||
The architecture plugin is chosen by the exact `(arch id, quant descriptor)` pair. A convert-time
|
||||
choice that must also hold at serve time is stated once with `--set`, which hands it to declare and
|
||||
writes it into the container. Storage format is not such a choice (the encoding is in the container
|
||||
and the plugin reads it there), but the **serving plugin** is: a `quantization_config.*` key
|
||||
re-derives the descriptor.
|
||||
|
||||
```sh
|
||||
rad-convert <bf16 Qwen3.8-27B> -o q38-27b-fp8.rad \
|
||||
--recipe data/recipes/q38-27b-fp8block-df2.recipe \
|
||||
--set quantization_config.quant_method=fp8 --set quantization_config.fmt=e4m3 \
|
||||
--set "quantization_config.weight_block_size=128 128" \
|
||||
--set quantization_config.activation_scheme=dynamic
|
||||
```
|
||||
|
||||
A bf16 checkpoint whose recipe writes Qwen's block FP8 must therefore be declared *as that format*;
|
||||
otherwise the selection pair does not match any plugin and the convert (or the boot) is refused. The
|
||||
same law bites the other way when a release's own `quant_method` names a pair no plugin claims.
|
||||
|
||||
## Inspect and validate the result
|
||||
|
||||
```sh
|
||||
rad-info out.rad # architecture, source quant, recipe, sections
|
||||
rad-info --recipe out.rad # the expanded rules — did the intended one win?
|
||||
rad-info -v out.rad | head -40 # every tensor, its encoding, and what made it
|
||||
rad-kbench -m out.rad --home $RADIANCE_HOME --report kbench.md
|
||||
radiance --model out.rad --debug-graph # every op resolves
|
||||
```
|
||||
|
||||
The converter's own error report (the worst `--report N` weights) measures each quantised weight
|
||||
against its source through the core decoder. `rad-kbench` adds the artifact test: weight operands
|
||||
read from the container and passed through each kernel's `layout`/`relayout` hooks. Both are on
|
||||
[[Radiance Tools]].
|
||||
|
||||
## Calibration (GPTQ, AWQ, AutoRound, ParoQuant)
|
||||
|
||||
`calib=DIR` is a directory an engine run wrote with `RADIANCE_CALIB_DIR` set: each layer's **Gram
|
||||
matrices of its input**, in the domain the kernel multiplies it in. A block that characterises an
|
||||
activation declares an f32 `[K,K]` PERSIST buffer and a `gram_accum` op; at `RADIANCE_CALIB_TOKENS`
|
||||
(default 65536) the run writes the files.
|
||||
|
||||
Three-step bootstrap: convert with plain `rtn` → serve that container with `RADIANCE_CALIB_DIR` →
|
||||
push real prompts through prefill → convert again with the Hessians. The second moment is robust to
|
||||
the first container's small quantisation error. A weight with no file is quantised uncalibrated and
|
||||
the converter says so; `awq`, `autoround` and `paroquant` fall back to the imatrix diagonal when one
|
||||
is given.
|
||||
|
||||
- The accumulator must be **PERSIST**: a transient buffer may share bytes and be overwritten between
|
||||
the launches it sums.
|
||||
- Files are per rank; the consumer sums them. Under expert parallelism a rank's `down` input covers
|
||||
only its experts' tokens; `gate_up` input is the full token set on every rank, so the sum scales
|
||||
`H` by world size — GPTQ is invariant to a positive scale.
|
||||
- Under tensor parallelism a column-split linear sees only its rank's columns, so a tap may write
|
||||
`gram.r<rank>.<stem>.part.bin`; libquant places the parts on the diagonal in rank order, leaving
|
||||
cross-rank products zero. They must cover K and be in the model's own domain.
|
||||
- `--imatrix` supplies the expert-popularity profile the placement planner ranks units by, the
|
||||
per-column importances a quantiser uses, and the error report's weighting.
|
||||
|
||||
## Reuse, drafters, merging
|
||||
|
||||
```sh
|
||||
# re-quantise only what changed
|
||||
rad-convert $SRC -o new.rad --recipe new.recipe --reuse old.rad --changed changed.txt
|
||||
|
||||
# merge a drafter shipped as its own repo (DFlash2, DSpark, MTP assistant)
|
||||
rad-convert $SRC -o out.rad --draft-model /srv/models/z-lab/Qwen3.8-27B-DFlash2 \
|
||||
--recipe data/recipes/q38-27b-mxfp4-df2.recipe
|
||||
|
||||
# merge a PEFT adapter at convert time
|
||||
rad-convert $SRC -o tuned.rad --lora ~/run/adapter --lora-strength 1.0 \
|
||||
--recipe data/recipes/q38-27b-mxfp4-df2.recipe
|
||||
```
|
||||
|
||||
- A drafter's tensors go in under `draft.`, its config under `draft.` keys. It has no embedding and
|
||||
no `lm_head` of its own; it uses the target's.
|
||||
- `--lora-map` exists for adapters trained on names the checkpoint does not use; a fused target
|
||||
(`qkv`) takes several targets and splits the adapter's rows in order.
|
||||
- `--add` + `--config` + `--tokenizer2` assemble a ComfyUI-style release of separate single files
|
||||
(Anima: diffusion model, text encoder, VAE). A name in two files is refused.
|
||||
|
||||
## `rad-export` — finetunes without hand-writing the command
|
||||
|
||||
```sh
|
||||
rad-export ~/outputs/my-run/checkpoint-400 --dry-run # what it would run
|
||||
rad-export ~/outputs/my-run/checkpoint-400 # adapter -> <name>.rad
|
||||
rad-export ~/exports/merged -o ~/models/mine.rad --recipe q38-27b-fp8block-df2
|
||||
```
|
||||
|
||||
| flag | default | meaning |
|
||||
|---|---|---|
|
||||
| `SRC` | required | a merged 16-bit checkpoint (`config.json` + `*.safetensors`) or a PEFT adapter (`adapter_config.json` + `adapter_model.safetensors`) |
|
||||
| `-o, --out FILE` | SRC's folder name + `.rad`, beside SRC or in `out_dir` | the container |
|
||||
| `--recipe NAME\|FILE\|auto` | `auto` | a recipe name, a file, or the one recipe whose header names the base model, takes a bf16 checkpoint and needs no calibration |
|
||||
| `--base DIR` | the adapter's `base_model_name_or_path`, from disk or the HF cache, else downloaded | the base a LoRA merges into |
|
||||
| `--draft DIR\|ORG/NAME` | the config, else the Hub repo the recipe's usage line names | the drafter when the recipe merges one; a usage line with a placeholder asks for `--draft` |
|
||||
| `--image IMAGE` | `$RADIANCE_IMAGE`, the config's, then `stilldeadcode/radiance` | run `rad-convert` in this image (docker, CPU-only, no network, **not pulled**) |
|
||||
| `--rad-convert PATH` | — | run this binary instead of docker; recipes come from its `share/radiance` |
|
||||
| `--memory M` | — | docker memory limit for the convert |
|
||||
| `--dry-run` | — | resolve recipe, drafter and paths; print the `rad-convert` line; write nothing |
|
||||
| `--keep-merged` | — | keep the merged checkpoint beside OUT |
|
||||
| `--config FILE` | `~/.config/rad-export/config.json` | the config below |
|
||||
| `--list-recipes [--json]` | — | list the recipes the image/installed tool carries |
|
||||
|
||||
- Input: a merged 16-bit checkpoint (`config.json` + `*.safetensors`) or a PEFT LoRA adapter
|
||||
(`adapter_config.json` + `adapter_model.safetensors`).
|
||||
- An adapter is merged **exactly as PEFT's `merge_and_unload`** computes it on CPU: `scale` =
|
||||
alpha/r (alpha/√r with rsLoRA), `rank_pattern`/`alpha_pattern` honoured, delta in fp32 rounded to
|
||||
the adapter's dtype. It streams tensor by tensor into 1 GiB shards, so a 27B merge needs a few GiB
|
||||
of RAM, not the model twice. Plain linear LoRA only: embedding LoRA, trained biases, DoRA, saved
|
||||
modules, convolution LoRA, a rank other than the config's, and a missing target are refused before
|
||||
anything is written. The adapter's tokeniser/template replace the base's where shipped.
|
||||
- The recipe comes from the image (`/opt/radiance/share/radiance/recipes`) or the installed
|
||||
`rad-convert`'s share; `auto` matches the base model's name in recipe headers and asks when several
|
||||
fit. A recipe needing calibration is listed but never picked by `auto` — a finetune's Hessians are
|
||||
not the stock model's.
|
||||
- It runs `rad-convert` on the CPU, under Docker as the caller with no network, and writes the log to
|
||||
`OUT.log`.
|
||||
- `~/.config/rad-export/config.json` pins what auto cannot know: `image`, `out_dir`,
|
||||
`recipes{model:recipe}`, `drafts{recipe:dir}`, `bases{repo:dir}`.
|
||||
- It is Python 3 with no dependencies; merging needs `torch`/`safetensors`, fetching a missing base or
|
||||
drafter needs `huggingface_hub`. The image carries no Python; extract the script with
|
||||
`docker run --rm --entrypoint busybox stilldeadcode/radiance cat /opt/radiance/bin/rad-export`.
|
||||
|
||||
## Scoring a container (the KL mode)
|
||||
|
||||
```sh
|
||||
# once, on the bf16 container
|
||||
radiance --model bf16.rad --kld-record /srv/kld/q38 --kld-corpus corpus.jsonl
|
||||
# each candidate, same flags otherwise
|
||||
radiance --model quant.rad --kld-ref /srv/kld/q38 --kld-out report.json
|
||||
```
|
||||
|
||||
- The corpus is JSONL, one `{"prompt", "score_from"?, "source"?}` a line; only `prompt` is required.
|
||||
Positions `[score_from, n-1)` are scored; `source` groups the report. Keep the scoring corpus
|
||||
disjoint from the calibration corpus.
|
||||
- `--kld-record` writes the reference's log-probabilities (f16 over the vocabulary, the 16 most
|
||||
likely tokens exactly, the token ids). DIR is never overwritten and is read only once `kld.json`
|
||||
exists, which is written last.
|
||||
- `--kld-ref` scores this model's f32 logits against stored rows: mean, median, 99/99.9/99.99th
|
||||
percentile KL, top-1 agreement, and both models' perplexity. `--kld-out` writes the report; `FILE.rows`
|
||||
is `[positions, 4]` f32 (KL, both NLLs, argmax agreement) for position-by-position comparison.
|
||||
- The mode turns speculation and the prefix cache off, turns the step pipeline off, and refuses an
|
||||
explicit spec depth above 0. The logits buffer holds a row per token of a step, so a large
|
||||
`--max-num-batched-tokens` costs pinned host memory as well as VRAM.
|
||||
- `--kld-lora NAME` scores every document through a served adapter, against a reference recorded
|
||||
from the merged checkpoint; `--kld-lora-sources PREFIX` runs only matching documents through it,
|
||||
the rest as the base, in the same steps — each document's rows should be byte-identical to the
|
||||
all-adapter and all-base runs.
|
||||
|
||||
The gate the fleet uses (mean KL ≤ 0.03, top-1 ≥ 95 %) is on [[frank-archon]] and
|
||||
[[Quantization Production]]; the mode here is only the instrument.
|
||||
|
||||
## Convert laws paid for in full
|
||||
|
||||
- **`--plan-only` first.** A plan that resolves every op will boot; it costs seconds against an hour
|
||||
of I/O on a large checkpoint.
|
||||
- **Declare runs at the step and context bounds, and some graphs depend on them.** Qwen4-Exp sizes
|
||||
its QSA buffers by `--max-ctx` and refuses a `--max-tok`/context pair whose score plane would pass
|
||||
2 GiB; the defaults (8192 tokens, the checkpoint's training context) fit most models, and a
|
||||
generator's largest pass may override `--max-tok`.
|
||||
- **Scale names follow the tensor.** `scale_name()` appends `_scale_inv` to the *tensor* name, so a
|
||||
stacked `…experts.gate_up_proj` keeps its scales at `…gate_up_proj_scale_inv`.
|
||||
- **File shadowing: first wins.** The reader opens index-named files sorted and emplaces every tensor
|
||||
each file physically holds, first-wins. A merged PLE shard must sort first
|
||||
(`00-ple.merged.safetensors`) or a raw shard wins the name.
|
||||
- **Never write through a staging symlink.** A checkpoint may symlink its raw index; `open(path,"w")`
|
||||
writes through it and corrupts the checkpoint. Write a temp file and `os.replace`.
|
||||
- **One encoding per expert table.** A whole expert kept bf16 is the only thing legal beside a
|
||||
table; a split tier (down=fp8, gate_up=w4) is unservable.
|
||||
- **The descriptor pair has no fallback** (above), and **convert is card-less**: GPTQ solves on CPU
|
||||
BLAS, so the GPU is irrelevant to production and the card only decides what can be decoded.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Radiance Model Families]] — the (arch id, quant descriptor) table and the command per model.
|
||||
- [[Radiance Plugins]] — quantiser and kernel-plugin contracts behind these options.
|
||||
- [[Radiance Tools]] — `rad-info`, `rad-kbench`, `rad-tune`.
|
||||
- [[Quantization Production]] — the fleet's calibration/KLD campaign and its gates.
|
||||
- [[frank-archon]] — the deployment whose recipes and convert laws these are.
|
||||
Reference in new issue
Block a user