serverAppConfigs/serverAppConfigs-docs/LLM/Radiance Operations.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
howto	Radiance Operations — running, reading, restarting	Running a radiance server: the container contract, host prerequisites, readiness, ledgers and safe restarts.	llm radiance operations howto	2026-10-30

Radiance operations

This is the engine-level runbook. The fleet's deployed shape, its unit and its deploy path live on frank-archon; the flags themselves on Radiance Serve Options.

Install

git clone https://codeberg.org/StillDeadcode/radiance.git && cd radiance
./build.sh                                   # configure, build for the cards present, run fast tests
./build.sh --no-hip                          # host backend only
cmake -S . -B build -G Ninja -DRAD_GPU_TARGETS='gfx1201'   # cards this machine lacks
cmake --install build                        # after -DCMAKE_INSTALL_PREFIX=/opt/radiance at first configure

Needs Linux, git, a C++20 compiler, CMake 3.21+, Ninja; for device kernels ROCm 10.1 or 7.2 (hipcc, ROCM_PATH if not /opt/rocm).

• Name the prefix at the first configure. The default plugin path <prefix>/share/radiance is compiled in from that configure; cmake --install build --prefix DIR alone gives a tree that looks for its plugins elsewhere.
• The prefix gets bin/, include/radiance/, lib/cmake/radiance/, libexec/radiance/, share/radiance/ (plugins and recipes) and share/doc/radiance/.
• With ROCm 10.1 the build turns VOPD pairing off because its LLVM 24 pairs in heap-address order, so two builds of one source can differ. The Docker image's compiler has the fix and keeps pairing (faster fp8 prefill GEMMs); declare such a compiler with -DRAD_HIP_VOPD_REPEATABLE=ON. ROCm 7.2 is unaffected.

Run without Docker:

export RADIANCE_HOME=$PWD/build/radiance_home   # plugins; an install finds its own
build/bin/radiance --model /srv/models/rad/minicpm5-2b-fp8.rad --port 8000 <flags>
build/bin/radiance --model StillDeadcode/minicpm5-2b-fp8 --port 8000 <flags>

--model also takes a repo id: the server runs scripts/hfget.sh into --download-dir, serves the one .rad at the repo's top level, or the directory as a checkpoint when it holds none. A repo with several containers is refused, naming them. Every start checks the hub; a start that cannot reach it serves the copy a complete earlier fetch left, and a transfer that fails part-way stops startup.

Container

The published image is stilldeadcode/radiance, kernels built for gfx1201. The host needs Docker and an amdgpu driver supporting ROCm 10.1 — not ROCm itself.

docker run -d --name radiance --network host --init --ulimit memlock=-1 \
    --device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined \
    --group-add "$(getent group render | cut -d: -f3)" \
    --group-add "$(getent group video | cut -d: -f3)" \
    -v /srv/models/rad:/models:ro \
    radiance --model /models/m.rad --tp 2 --host :: --port 8000

Each option is load-bearing:

option	why
--device /dev/kfd --device /dev/dri, render/video groups	the cards
--security-opt seccomp=unconfined	Docker's default profile refuses io_uring (a disk-tier n-gram table's reads) and the NUMA calls the ROCm runtime makes for host memory
--init	an init process as PID 1 forwards SIGTERM; the engine finishes its step and releases the cards
--ulimit memlock=-1	a host-side pinned n-gram copy can be tens of GiB, far past Docker's default
--network host	the server binds the host's addresses directly, IPv4 and IPv6, no NAT

The image is FROM scratch around the stripped install tree: busybox as the only shell, plus curl, jq and a CA bundle for hfget.sh; no Python, no package manager. Tools (rad-convert, rad-info, rad-kbench, rad-tune, rad-schemas, rad-tokcheck) are on PATH; recipes are in /opt/radiance/share/radiance/recipes/. Its entrypoint is radiance. It sets GPU_MAX_ALLOC_PERCENT=100 and GPU_MAX_HEAP_SIZE=100 because the HIP runtime's defaults cap one allocation and the heap below what the placement plan hands out.

Build your own with docker/build.sh (-t 'gfx1200;gfx1201' for targets, -j compile jobs, -E to skip tests, -s host to build remotely). The context is git archive of the commit, so the image holds only committed content. The rootfs stage fails the build if any ELF cannot be resolved inside a chroot of the tree.

Compose: one file per model plus common.yaml, copied to the serving host with an .env setting the image, models directory (mounted read-only at /models), API key, port, user and GPU groups. Every model listens on the same port, and most use both cards, so run one at a time. The models directory must be on a local filesystem that does direct I/O (ext4, XFS, btrfs); on btrfs with compression, a container copied into a chattr +C directory is stored uncompressed and direct reads work.

A plugin of your own is mounted and put first on the search path; nothing in the image changes:

-v /srv/radiance-plugins:/plugins:ro -e RADIANCE_HOME=/plugins:/opt/radiance/share/radiance

Host prerequisites (upstream's two-R9700 checklist)

item	setting
firmware	Above 4G Decoding on (Resizable BAR needs it); CSM off (pure UEFI)
kernel command line	iommu=pt amdgpu.ppfeaturemask=0xfff7ffff amdgpu.ras_enable=0
power/clock	OverDrive bit needed for ppfeaturemask; apply a power cap only while no process holds /dev/kfd
memlock	--ulimit memlock=-1 (container) / LimitMEMLOCK=infinity (systemd) when host memory is pinned
two engines, one tmpfs table	amdgpu.no_system_mem_limit=1, or the second pin can be refused
storage	a local fs with O_DIRECT for the container; a tmpfs sized for the table for --ngram-placement ram=DIR

Why each: iommu=pt keeps DMA identity so the cards can reach each other's memory and pinned host pages; ppfeaturemask enables the power controls; ras_enable=0 turns VRAM ECC off (ECC reserves about 2 GiB a card). Verify peer access with a rate in lspci/the card tables — a missing cell means no peer access; check iommu=pt and the BAR size. The fleet's own boot-line law for Dankclaw (iommu=pt, tracked in the repo) is on dankclaw-iommu-translation-off; Dankserver's cards are a different box and its posture is on Dankserver System Specs.

Readiness: the receipt, then /health

Startup states every budget before anything is allocated, then loads. /health (and /v1/health, /ping) answers only once the load completes, so a bare health gate is honest. While it loads, expect 503 for minutes; do not restart over it. --live draws the live view (redirected, one greppable key=value line). /diagnostics returns the startup report as JSON — same-machine-only without an API key — with every option and its source and the hardware read again now.

Read the receipt in this order:

1. elastic — what the experts and KV cache share after non-expert weights.
2. The split: experts … / kv cache …, and unclaimed if any.
3. kv: the pool backs N % of the worst case … M full-length sessions — the referee for whether the demand signal (--max-model-len × --max-num-seqs) is funded.
4. activation arena and what each size lends; prefill staging; hardware queues and their VRAM.
5. host_pool opened/used, vram_weights, the PLE or n-gram placement, the vision tower's placement.
6. device: the N cards reach each other's memory directly or the failed pairs; expert prefetch on/off; chunk geometry; speculation:; chained launches; draft trees.
7. card N settled at … free — memory taken after the budget read comes out of headroom; a warning names the --gpu-headroom-mib that clears it.

A boot refusal prints the reason and exits nonzero; when it happens inside podman run, the container stops at once and the unit's restart policy re-runs the same refusal. A unit in activating (auto-restart) with a fast nonzero exit is therefore a preflight/refusal or a boot refusal, not an engine fault: read the journal line and the argv, not the GPU.

Lifecycle and restart

• SIGTERM finishes the current step and releases the cards. In a container, --init forwards it; podman stop -t N/docker stop is the graceful path.
• A systemd unit should exec podman wait (or run in the foreground) so the script's lifetime is the container's: a crash is a nonzero exit and the restart policy is real. Set TimeoutStopSec above the stop grace, or systemd records Failed with result timeout on a healthy stop.
• Restart=always rather than on-failure for a server whose front end can exit cleanly after a worker fault.
• Model load is slow (minutes cold). The deployer's LLM wait budget is on Deployment Flow; a container restart loop during a legitimate load is worse than waiting.

Two engines on one set of cards

• ROCR_VISIBLE_DEVICES picks each engine's cards explicitly.
• --weight-prefetch off: a weight read beside a collective puts a queue switch into every decode step when another server shares the cards.
• --ngram-placement ram=DIR: two engines on separate card sets hold one table in host memory (DIR/ngram-<id>.tab in a tmpfs, MAP_SHARED, pinned). The <id> hashes the container path, size, mtime and the table's place, so a file never serves another model. The first engine fills it; a second verifies samples and only pins it; the last to leave deletes it. The driver counts a shared page against every process that pins it, so the pinned total is twice the table.
• The engine prefers discrete cards (all of the first one's architecture), integrated only when there is nothing else; on integrated graphics device memory is host memory and the budget leaves an eighth of host RAM to the host.

Logs and diagnostics

• -v / -vv; RADIANCE_LOG sets the level without a flag. RADIANCE_LOG_STEPS=1 prints every step's token count (how a chunk geometry is confirmed in practice).
• The engine writes to stderr. Under a container, the journal holds only the container's own output; a rootless podman logs covers only the current instance across recreations.
• /metrics is Prometheus text: engine steps by kind and their seconds, decode/prefill rates, prefill chunks held back, steps held by interleave, draft acceptance and depth, prompt-lookup proposals held, KV bytes and the tokens they address, stream misses and bytes, prefix hits and evictions, preemptions and queue depth. /stats, /graph, /sessions back the dashboard. scripts/statsdump.py records /stats to JSONL with time and gap, for a machine with no scraper (its key comes from --api-key or RADIANCE_API_KEY).
• RADIANCE_PROFILE_EVERY, RADIANCE_DUMP_BUF/RADIANCE_DUMP_DIR, RADIANCE_OP_ORACLE(+) and the RADIANCE_DEBUG_* family are debug instruments; they are not deployment settings.

Verifying a deploy (fleet law)

deploy.sh green is not proof of what is running. A leftover systemd drop-in or a stale container identity can serve a different model at a different shape while /health is 200. After deploying any recently-benched service:

1. systemctl show <svc> -p DropInPaths — must be empty.
2. tr '\0' ' ' </proc/$(pidof radiance)/cmdline (or the container's Config.Cmd) — the argv you meant.
3. The boot ledger — the shape you meant.
4. A behavioural probe — /v1/models for the served id, one chat turn, one tool call, and a vision probe with a solid colour and an exact expected word.

/v1/models is an echo on some runtimes; on radiance it is authoritative for the alias, but the argv and the ledger are what prove the payload and shape. The three-pin revert discipline for a payload swap is on Model Swap and Rollback; the live deployment is frank-archon.

Security posture

• --api-key requires a bearer token or X-Api-Key on every route except /health, /v1/health, /ping and the dashboard; the key is a command-line argument, so it is readable in /proc/<pid>/cmdline by any local user of that host. On a single-tenant inference box the network boundary is the real control; do not run one where untrusted local users share the host.
• /diagnostics is same-machine-only without a key, because it prints the whole configuration.
• --security-opt seccomp=unconfined is a genuine widening, required by io_uring and the ROCm runtime's NUMA calls; it belongs on a dedicated model host, not a general-purpose one.
• Mount model directories read-only; a container with /models writable adds nothing a serve needs.
• A control vector can remove refusal behaviour. Loading one is a safety decision, not a performance one; the flag is --control-vector, and --control-vector disabled is the default.

Failure signatures at startup

what you see	what it is
unknown/retired flag, or a flag refused by name	a boot refusal — the pin does not have it; change the trio, not the invocation
"no plugin claims <arch> at <quant>; served at: …"	the (id, descriptor) pair has no plugin; fix the descriptor with --set/convert
"reports ABI N, engine M"	rebuild the plugin against this engine
budget refusal / host_pool short	the shape does not fit this card/host; lower demand or pin more host RAM
--ngram-placement ram refuses	memlock short; raise it or use auto/disk
--kv-cache-dtype fp8_v4/k6_v4 refused, plugin named	no kernels for this geometry
--p2p on refuses, pairs named	the driver has no peer access; use auto/off
first launch of a never-run kernel faults	too little --gpu-headroom-mib for its code object
unit loop, no container, no engine log	a preflight refusal (TTM/memlock/digest/model) re-run by Restart=always

Related

• Radiance Serve Options — every knob and its law.
• Radiance HTTP API — readiness and the wire.
• Radiance Tools — how to corroborate a receipt.
• frank-archon — the fleet's radiance unit, guard chain and Eras.
• Model Swap and Rollback — the swap discipline this page's verification serves.
• Deployment Flow — how a committed change reaches the host.