docs(radiance): reference the engine's options, plugins and conversion
Nine pages under LLM/ document the upstream radiance engine at 8da043c: - engine shape, plugin model, RAD2 container, ABI and environment - every serve flag grouped with defaults and the law behind each knob - the HTTP surface, request fields and refusal classes - rad-convert, the recipe language, quantisers, imports, reuse, rad-export, KL scoring - the (arch id, quant) family table and the recipe of record per model - kernel/quantiser plugin authoring, encodings, build and checks - architecture plugin authoring: declare, step, probe, lane and media rules - rad-info/radbench/radtune and the shell harnesses - the container contract, host prerequisites, readiness and restart triage Also completes the dangling Related line on the frank-archon endpoint page.
This commit is contained in:
Frank committed
1 parent
da5eb968f9
commit
3a04e34414
12 files changed
+2319
-2
No files matched your search
@@ -0,0 +1,239 @@
|
||||
---
|
||||
type: "howto"
|
||||
title: "Radiance Operations — running, reading, restarting"
|
||||
description: "Running a radiance server: the container contract, host prerequisites, readiness, ledgers and safe restarts."
|
||||
tags:
|
||||
- llm
|
||||
- radiance
|
||||
- operations
|
||||
- howto
|
||||
timestamp: 2026-10-30
|
||||
---
|
||||
|
||||
# Radiance operations
|
||||
|
||||
This is the engine-level runbook. The fleet's deployed shape, its unit and its deploy path live on
|
||||
[[frank-archon]]; the flags themselves on [[Radiance Serve Options]].
|
||||
|
||||
## Install
|
||||
|
||||
```sh
|
||||
git clone https://codeberg.org/StillDeadcode/radiance.git && cd radiance
|
||||
./build.sh # configure, build for the cards present, run fast tests
|
||||
./build.sh --no-hip # host backend only
|
||||
cmake -S . -B build -G Ninja -DRAD_GPU_TARGETS='gfx1201' # cards this machine lacks
|
||||
cmake --install build # after -DCMAKE_INSTALL_PREFIX=/opt/radiance at first configure
|
||||
```
|
||||
|
||||
Needs Linux, git, a C++20 compiler, CMake 3.21+, Ninja; for device kernels ROCm 10.1 or 7.2
|
||||
(`hipcc`, `ROCM_PATH` if not `/opt/rocm`).
|
||||
|
||||
- **Name the prefix at the first configure.** The default plugin path `<prefix>/share/radiance` is
|
||||
compiled in from that configure; `cmake --install build --prefix DIR` alone gives a tree that looks
|
||||
for its plugins elsewhere.
|
||||
- The prefix gets `bin/`, `include/radiance/`, `lib/cmake/radiance/`, `libexec/radiance/`,
|
||||
`share/radiance/` (plugins and recipes) and `share/doc/radiance/`.
|
||||
- With ROCm 10.1 the build turns VOPD pairing off because its LLVM 24 pairs in heap-address order, so
|
||||
two builds of one source can differ. The Docker image's compiler has the fix and keeps pairing
|
||||
(faster fp8 prefill GEMMs); declare such a compiler with `-DRAD_HIP_VOPD_REPEATABLE=ON`. ROCm 7.2 is
|
||||
unaffected.
|
||||
|
||||
Run without Docker:
|
||||
|
||||
```sh
|
||||
export RADIANCE_HOME=$PWD/build/radiance_home # plugins; an install finds its own
|
||||
build/bin/radiance --model /srv/models/rad/minicpm5-2b-fp8.rad --port 8000 <flags>
|
||||
build/bin/radiance --model StillDeadcode/minicpm5-2b-fp8 --port 8000 <flags>
|
||||
```
|
||||
|
||||
`--model` also takes a repo id: the server runs `scripts/hfget.sh` into `--download-dir`, serves the
|
||||
one `.rad` at the repo's top level, or the directory as a checkpoint when it holds none. A repo with
|
||||
several containers is refused, naming them. Every start checks the hub; a start that cannot reach it
|
||||
serves the copy a complete earlier fetch left, and a transfer that fails part-way stops startup.
|
||||
|
||||
## Container
|
||||
|
||||
The published image is `stilldeadcode/radiance`, kernels built for `gfx1201`. The host needs Docker
|
||||
and an `amdgpu` driver supporting ROCm 10.1 — **not** ROCm itself.
|
||||
|
||||
```sh
|
||||
docker run -d --name radiance --network host --init --ulimit memlock=-1 \
|
||||
--device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined \
|
||||
--group-add "$(getent group render | cut -d: -f3)" \
|
||||
--group-add "$(getent group video | cut -d: -f3)" \
|
||||
-v /srv/models/rad:/models:ro \
|
||||
radiance --model /models/m.rad --tp 2 --host :: --port 8000
|
||||
```
|
||||
|
||||
Each option is load-bearing:
|
||||
|
||||
| option | why |
|
||||
|---|---|
|
||||
| `--device /dev/kfd --device /dev/dri`, `render`/`video` groups | the cards |
|
||||
| `--security-opt seccomp=unconfined` | Docker's default profile refuses io_uring (a disk-tier n-gram table's reads) and the NUMA calls the ROCm runtime makes for host memory |
|
||||
| `--init` | an init process as PID 1 forwards SIGTERM; the engine finishes its step and releases the cards |
|
||||
| `--ulimit memlock=-1` | a host-side pinned n-gram copy can be tens of GiB, far past Docker's default |
|
||||
| `--network host` | the server binds the host's addresses directly, IPv4 and IPv6, no NAT |
|
||||
|
||||
The image is `FROM scratch` around the stripped install tree: busybox as the only shell, plus `curl`,
|
||||
`jq` and a CA bundle for `hfget.sh`; **no Python, no package manager**. Tools (`rad-convert`,
|
||||
`rad-info`, `rad-kbench`, `rad-tune`, `rad-schemas`, `rad-tokcheck`) are on `PATH`; recipes are in
|
||||
`/opt/radiance/share/radiance/recipes/`. Its entrypoint is `radiance`. It sets
|
||||
`GPU_MAX_ALLOC_PERCENT=100` and `GPU_MAX_HEAP_SIZE=100` because the HIP runtime's defaults cap one
|
||||
allocation and the heap below what the placement plan hands out.
|
||||
|
||||
Build your own with `docker/build.sh` (`-t 'gfx1200;gfx1201'` for targets, `-j` compile jobs, `-E`
|
||||
to skip tests, `-s host` to build remotely). The context is `git archive` of the commit, so the image
|
||||
holds only committed content. The `rootfs` stage fails the build if any ELF cannot be resolved inside
|
||||
a chroot of the tree.
|
||||
|
||||
Compose: one file per model plus `common.yaml`, copied to the serving host with an `.env` setting the
|
||||
image, models directory (mounted read-only at `/models`), API key, port, user and GPU groups. Every
|
||||
model listens on the same port, and most use both cards, so **run one at a time**. The models
|
||||
directory must be on a local filesystem that does direct I/O (ext4, XFS, btrfs); on btrfs with
|
||||
compression, a container copied into a `chattr +C` directory is stored uncompressed and direct reads
|
||||
work.
|
||||
|
||||
A plugin of your own is mounted and put first on the search path; nothing in the image changes:
|
||||
|
||||
```sh
|
||||
-v /srv/radiance-plugins:/plugins:ro -e RADIANCE_HOME=/plugins:/opt/radiance/share/radiance
|
||||
```
|
||||
|
||||
## Host prerequisites (upstream's two-R9700 checklist)
|
||||
|
||||
| item | setting |
|
||||
|---|---|
|
||||
| firmware | Above 4G Decoding on (Resizable BAR needs it); CSM off (pure UEFI) |
|
||||
| kernel command line | `iommu=pt amdgpu.ppfeaturemask=0xfff7ffff amdgpu.ras_enable=0` |
|
||||
| power/clock | OverDrive bit needed for `ppfeaturemask`; apply a power cap only while no process holds `/dev/kfd` |
|
||||
| memlock | `--ulimit memlock=-1` (container) / `LimitMEMLOCK=infinity` (systemd) when host memory is pinned |
|
||||
| two engines, one tmpfs table | `amdgpu.no_system_mem_limit=1`, or the second pin can be refused |
|
||||
| storage | a local fs with O_DIRECT for the container; a tmpfs sized for the table for `--ngram-placement ram=DIR` |
|
||||
|
||||
Why each: `iommu=pt` keeps DMA identity so the cards can reach each other's memory and pinned host
|
||||
pages; `ppfeaturemask` enables the power controls; `ras_enable=0` turns VRAM ECC off (ECC reserves
|
||||
about 2 GiB a card). Verify peer access with a rate in `lspci`/the card tables — a missing cell means
|
||||
no peer access; check `iommu=pt` and the BAR size. The fleet's own boot-line law for Dankclaw
|
||||
(`iommu=pt`, tracked in the repo) is on [[dankclaw-iommu-translation-off]]; Dankserver's cards are a
|
||||
different box and its posture is on [[Dankserver System Specs]].
|
||||
|
||||
## Readiness: the receipt, then `/health`
|
||||
|
||||
Startup states every budget **before anything is allocated**, then loads. `/health` (and
|
||||
`/v1/health`, `/ping`) answers only once the load completes, so a bare health gate is honest. While it
|
||||
loads, expect 503 for minutes; do not restart over it. `--live` draws the live view (redirected, one
|
||||
`greppable key=value` line). `/diagnostics` returns the startup report as JSON — **same-machine-only
|
||||
without an API key** — with every option and its source and the hardware read again now.
|
||||
|
||||
Read the receipt in this order:
|
||||
|
||||
1. `elastic` — what the experts and KV cache share after non-expert weights.
|
||||
2. The split: `experts … / kv cache …`, and `unclaimed` if any.
|
||||
3. `kv: the pool backs N % of the worst case … M full-length sessions` — the referee for whether the
|
||||
demand signal (`--max-model-len` × `--max-num-seqs`) is funded.
|
||||
4. `activation arena` and what each size lends; `prefill staging`; `hardware queues` and their VRAM.
|
||||
5. `host_pool` opened/used, `vram_weights`, the PLE or n-gram placement, the vision tower's placement.
|
||||
6. `device: the N cards reach each other's memory directly` or the failed pairs;
|
||||
`expert prefetch on/off`; `chunk geometry`; `speculation:`; `chained launches`; `draft trees`.
|
||||
7. `card N settled at … free` — memory taken after the budget read comes out of headroom; a warning
|
||||
names the `--gpu-headroom-mib` that clears it.
|
||||
|
||||
A boot refusal prints the reason and exits nonzero; when it happens inside `podman run`, the
|
||||
container stops at once and the unit's restart policy re-runs the same refusal. A unit in
|
||||
`activating (auto-restart)` with a fast nonzero exit is therefore a **preflight/refusal or a boot
|
||||
refusal**, not an engine fault: read the journal line and the argv, not the GPU.
|
||||
|
||||
## Lifecycle and restart
|
||||
|
||||
- SIGTERM finishes the current step and releases the cards. In a container, `--init` forwards it;
|
||||
`podman stop -t N`/`docker stop` is the graceful path.
|
||||
- A systemd unit should `exec podman wait` (or run in the foreground) so the script's lifetime is the
|
||||
container's: a crash is a nonzero exit and the restart policy is real. Set `TimeoutStopSec` above
|
||||
the stop grace, or systemd records `Failed with result timeout` on a healthy stop.
|
||||
- `Restart=always` rather than `on-failure` for a server whose front end can exit cleanly after a
|
||||
worker fault.
|
||||
- Model load is slow (minutes cold). The deployer's LLM wait budget is on [[Deployment Flow]]; a
|
||||
container restart loop during a legitimate load is worse than waiting.
|
||||
|
||||
## Two engines on one set of cards
|
||||
|
||||
- `ROCR_VISIBLE_DEVICES` picks each engine's cards explicitly.
|
||||
- `--weight-prefetch off`: a weight read beside a collective puts a queue switch into every decode
|
||||
step when another server shares the cards.
|
||||
- `--ngram-placement ram=DIR`: two engines on separate card sets hold one table in host memory
|
||||
(`DIR/ngram-<id>.tab` in a tmpfs, `MAP_SHARED`, pinned). The `<id>` hashes the container path, size,
|
||||
mtime and the table's place, so a file never serves another model. The first engine fills it; a
|
||||
second verifies samples and only pins it; the last to leave deletes it. The driver counts a shared
|
||||
page against every process that pins it, so the pinned total is twice the table.
|
||||
- The engine prefers discrete cards (all of the first one's architecture), integrated only when
|
||||
there is nothing else; on integrated graphics device memory is host memory and the budget leaves an
|
||||
eighth of host RAM to the host.
|
||||
|
||||
## Logs and diagnostics
|
||||
|
||||
- `-v` / `-vv`; `RADIANCE_LOG` sets the level without a flag. `RADIANCE_LOG_STEPS=1` prints every
|
||||
step's token count (how a chunk geometry is confirmed in practice).
|
||||
- The engine writes to stderr. Under a container, the journal holds only the container's own output;
|
||||
a rootless `podman logs` covers only the current instance across recreations.
|
||||
- `/metrics` is Prometheus text: engine steps by kind and their seconds, decode/prefill rates,
|
||||
prefill chunks held back, steps held by interleave, draft acceptance and depth, prompt-lookup
|
||||
proposals held, KV bytes and the tokens they address, stream misses and bytes, prefix hits and
|
||||
evictions, preemptions and queue depth. `/stats`,
|
||||
`/graph`, `/sessions` back the dashboard. `scripts/statsdump.py` records `/stats` to JSONL with
|
||||
time and gap, for a machine with no scraper.
|
||||
- `RADIANCE_PROFILE_EVERY`, `RADIANCE_DUMP_BUF`/`RADIANCE_DUMP_DIR`, `RADIANCE_OP_ORACLE`(+) and the
|
||||
`RADIANCE_DEBUG_*` family are debug instruments; they are not deployment settings.
|
||||
|
||||
## Verifying a deploy (fleet law)
|
||||
|
||||
`deploy.sh` green is not proof of what is running. A leftover systemd drop-in or a stale container
|
||||
identity can serve a different model at a different shape while `/health` is 200. After deploying
|
||||
any recently-benched service:
|
||||
|
||||
1. `systemctl show <svc> -p DropInPaths` — must be empty.
|
||||
2. `tr '\0' ' ' </proc/$(pidof radiance)/cmdline` (or the container's `Config.Cmd`) — the argv you
|
||||
meant.
|
||||
3. The boot ledger — the shape you meant.
|
||||
4. A behavioural probe — `/v1/models` for the served id, one chat turn, one tool call, and a vision
|
||||
probe with a solid colour and an exact expected word.
|
||||
|
||||
`/v1/models` is an echo on some runtimes; on radiance it is authoritative for the alias, but the
|
||||
argv and the ledger are what prove the payload and shape. The three-pin revert discipline for a
|
||||
payload swap is on [[Model Swap and Rollback]]; the live deployment is [[frank-archon]].
|
||||
|
||||
## Security posture
|
||||
|
||||
- `--api-key` requires a bearer token or `X-Api-Key` on every route except `/health`, `/v1/health`,
|
||||
`/ping` and the dashboard; the key is a command-line argument, so it is readable in
|
||||
`/proc/<pid>/cmdline` by any local user of that host. On a single-tenant inference box the network
|
||||
boundary is the real control; do not run one where untrusted local users share the host.
|
||||
- `/diagnostics` is same-machine-only without a key, because it prints the whole configuration.
|
||||
- `--security-opt seccomp=unconfined` is a genuine widening, required by io_uring and the ROCm
|
||||
runtime's NUMA calls; it belongs on a dedicated model host, not a general-purpose one.
|
||||
- Mount model directories read-only; a container with `/models` writable adds nothing a serve needs.
|
||||
- A control vector can remove refusal behaviour. Loading one is a safety decision, not a
|
||||
performance one; the flag is `--control-vector`, and `--control-vector disabled` is the default.
|
||||
|
||||
## Failure signatures at startup
|
||||
|
||||
| what you see | what it is |
|
||||
|---|---|
|
||||
| unknown/retired flag, or a flag refused by name | a boot refusal — the pin does not have it; change the trio, not the invocation |
|
||||
| "no plugin claims `<arch>` at `<quant>`; served at: …" | the (id, descriptor) pair has no plugin; fix the descriptor with `--set`/convert |
|
||||
| "reports ABI N, engine M" | rebuild the plugin against this engine |
|
||||
| budget refusal / `host_pool` short | the shape does not fit this card/host; lower demand or pin more host RAM |
|
||||
| `--ngram-placement ram` refuses | memlock short; raise it or use `auto`/`disk` |
|
||||
| `--kv-cache-dtype fp8_v4`/`k6_v4` refused, plugin named | no kernels for this geometry |
|
||||
| `--p2p on` refuses, pairs named | the driver has no peer access; use `auto`/`off` |
|
||||
| first launch of a never-run kernel faults | too little `--gpu-headroom-mib` for its code object |
|
||||
| unit loop, no container, no engine log | a preflight refusal (TTM/memlock/digest/model) re-run by `Restart=always` |
|
||||
|
||||
## Related
|
||||
|
||||
- [[Radiance Serve Options]] — every knob and its law.
|
||||
- [[Radiance HTTP API]] — readiness and the wire.
|
||||
- [[Radiance Tools]] — how to corroborate a receipt.
|
||||
- [[frank-archon]] — the fleet's radiance unit, guard chain and Eras.
|
||||
- [[Model Swap and Rollback]] — the swap discipline this page's verification serves.
|
||||
- [[Deployment Flow]] — how a committed change reaches the host.
|
||||
Reference in new issue
Block a user