Files

T

Frank 0f9faf05ea radiance runbook: MNBT A/B verdict + rad-tune tune-dir bind-mount

Snapshot of a parallel session's completed docs edit (mtime 03:07,
was blocking the deploy tree-clean gate): --max-num-batched-tokens
2048 A/B-verified (231/460/448 t/s cold prefill ladder), rad-tune
cache survives via FRANK_PRIMARY_STATE_DIR/tune bind-mount
(36 GEMM rows, generated 2026-10-06).

2026-10-06 03:08:30 -07:00

20 KiB

Raw Blame History

frank-primary — Radiance (StillDeadcode), Qwen3.8-Flash-Next-FP8 NATIVE, 2× R9700 TP=2

Status: ACTIVE 2026-10-05 → · Host: Dankserver (192.168.0.235) · Port: 7770 · GPU: 2× Radeon AI PRO R9700 (gfx1201), TP=2

The FreeToken PR #587 endpoint was replaced by the radiance engine (C++/HIP, libr4d kernels for gfx1201, no Python in the serving path) driving the digest-pinned image stilldeadcode/radiance@sha256:4c3f484564bb2872784e4… (v1.0.7 line, 287 MiB, libr4d@0.5.0) as a plain podman run through scripts/serve-frank-primary.sh under the root systemd unit (frank-primary.service, User=dank, LimitMEMLOCK=infinity). Same port, same served alias (frank-primary), same container identity (frank-primary = the unit name). Rollback: git tag frank-primary-freetoken-pr587-20261005 (FreeToken trio + Dockerfile), image localhost/frank-primary-freetoken:pr587-77a7201, payload /home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8.

Why radiance

TP=2 is real here. FreeToken's TP=2 dies in the pynccl.cu ROCm JIT (cuda_runtime.h unresolvable for gfx1201 — probed 2026-10-06); the second R9700 idled. Radiance's TP wire is libr4d's own one-shot all-reduce (no RCCL), legal on this board since the 2026-09-20 swap (ProArt B850, x8/x8 on CPU root ports, Routing+/ReqEn+ everywhere).
Native FP8 end to end (Frank's call): the official Qwen/Qwen3.8-Flash-Next-FP8 checkpoint keeps its modelopt block-FP8 encoding — no requantisation, no calibration run, no 4-bit codebook.
Every tier is explicit and measured, not an estimator: placement expert_tiered with a pinned host pool, --ngram-placement ram for the 47.7 GiB PLE table, --expert-vs-cache-ratio 0.82 (the publisher's measured 99 %-hit point; 0.87 already collapses to 0 %), fp8 KV.

The payload: native FP8, reshaped then converted (one recipe rule)

Raw checkpoint: /home/dank/models/Qwen/Qwen3.8-Flash-Next-FP8 (ungated repo, 172.8 GB, 131 shards + index + tokenizer + vision configs). Fetched by scripts/download-frank-primary.sh (vendored hfget.sh, curl+jq, no HF token needed) and verified byte-for-byte: all 131 shards sha256-match the Hub LFS oids (2026-10-05, log /home/dank/logs/verify-qwen38-official-fp8.log).
The container: /home/dank/models/rad/qwen4exp-fp8-native.rad (177,109 MiB measured = 173.0 GiB; floor FRANK_PRIMARY_RAD_MIN_MIB=177000 in the env gates a truncated convert). Built by scripts/build-frank-primary-rad.sh:
1. stage /home/dank/models/stage-qwen4exp-fp8 — config.json with quantization_config dropped + relative symlinks into the raw checkout;
2. stack-qwen38-official-fp8.py reshapes the staging dir — two transforms, both value-preserving (script docstring carries the full argument): the plugin maps the STACKED expert layout (mlp.experts.gate_up_proj [512,1280,2560] + _scale_inv at the same index) while the official repo ships per-expert unfused tensors, so staging byte-stacks all 49 stacks' (48 trunk + MTP) 512 experts with NO requantisation; and the PLE table ships as 128 scaleless E4M3 shards + one bf16 scalar (1.9931793212890625e-04) while the plugin's fp8 branch wants the codes beside a ONE-ELEMENT bf16 scale plane — so staging decodes to a single bf16 shard_0 with the checkpoint's own scale (e4m3 decode and the scale are both exact in bf16);
3. rad-convert with ONE --quant rule — pure copy for everything else (every weight keeps the checkpoint's block-FP8 planes: X.weight E4M3 codes + X.weight_scale_inv bf16 [128×128] scales) — and the rtn rule re-encodes the bf16 PLE table at the same fixed scale, an exact round-trip (nearest-code distance 2⁻³ vs the bf16 rounding perturbation 2⁻⁹): the container gets the checkpoint's own PLE codes plus the one-element scale plane (blk.1.ple_ngram.scale, view {1});
4. THREE LAWS the live runs proved (each cost a full verify+declare cycle; all fixed in the two scripts, 2026-10-05/06):
  - Scale-name law: scale_name() appends _scale_inv to the TENSOR NAME verbatim — a stacked tensor named …experts.gate_up_proj (no .weight suffix) has its scale plane at …experts.gate_up_proj_scale_inv, NOT …weight_scale_inv.
  - File-shadow law: the reader opens every index-named file in SORTED order and emplaces EVERY tensor each file physically holds, first-wins (checkpoint.cpp add_shard). The raw model-000NN shards carry tensors NAMED exactly what the merged PLE shard is named, so the merged file MUST sort first (00-ple.merged.safetensors) or a raw fp8 shard wins the name and the plan reads one shard as the codes.
  - Symlink-through law: staging symlinks the raw index into the stage; open(path,"w") writes THROUGH it and corrupts the raw checkpoint (raw verify then fails 143/144). The stack script writes index/config temp+os.replace — an atomic swap replaces the symlink itself; the build script COPIES the index and symlinks only the shards.
5. rad-info summary + size floor.
Why the staging dir is mandatory (the descriptor law, spec §2.4): the plugin is selected by the exact (arch_id, quant descriptor) pair with no fallback. A raw official checkpoint carries quantization_config.quant_method: "fp8" → descriptor "fp8" → no shipped plugin claims (qwen4exp, "fp8") — raw serve AND unmodified convert both refuse. The shipped qwen4exp_fp8 plugin claims (qwen4exp, "") (recipe-built containers). rad_convert.cpp writes w.set_quant(meta.quant) into the header, so the staged config's empty quant method flows through as the container descriptor while every weight stays FP8 data. Native precision is untouched; only the selection key differs.
--plan-only first: the convert's declare phase is the engine's own, so a plan that resolves every op will boot. The convert runs card-less (declare = CPU-side layout queries); add --device only if a plugin load ever demands it.

The shape (`scripts/frank-primary.env`, every value commented there)

Knob	Value	Why
Image	`stilldeadcode/radiance@sha256:4c3f4845…`	digest-pinned INDEX digest (RepoDigests also carry the per-arch manifest `1c582005…`; the image ID `69d05724…` is the config digest, NOT a manifest digest — pinning it sends `podman run` to the network); entrypoint = engine, tools (`rad-convert`/`rad-info`/`rad-tune`/`rad-kbench`) on PATH
Container	`frank-primary`	identity = service name across runtimes; legacy list retires `frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe`
`--tp 2 --tp-wire wht6`	both R9700s	fused WHT-6 lossy wire: +5.6–6.3 % prefill, ~−1.3 % 8-stream decode at TP2; `--tp-wire-min-kb` stays engine default (128 KiB)
`--max-num-seqs 2`	first boot	Frank: "parallelism = 2 to start". Raise only after a real-traffic look at slot occupancy + the ledger's KV headroom line. The fused all-reduce row count is `seqs × (1 + spec) = 8`
`--max-model-len 200000`	served window	native 262,144; the publisher's production value and what `models.yml` quotes. Longer prompt = refused, never truncated
`--placement expert_tiered --host-pool-mib 40960`	pinned expert pool per rank	native FP8 experts are 112.8 GiB total (56.4 GiB/rank slice) vs ~29 GiB at the 4-bit publisher shape — the pool is the price of native precision. Resize from the boot ledger's coverage line, never arithmetic
`--gpu-headroom-mib 96 --expert-vs-cache-ratio 0.82`	publisher values	0.82 = measured 99 % expert-cache hit; 0.87 → 0 % (the cliff)
`--kv-cache-dtype fp8`	7.32 KiB/token	12 QSA full-attention layers, 2 KV heads × 256 dim, K+V; a 200K-token sequence is 1.43 GiB. KV is sized from the VRAM the ratio leaves
`--num-speculative-tokens 3`	MTP head	the checkpoint carries `mtp.*` (3,101 tensors, `mtp_num_hidden_layers: 1`); the plugin's probe returns draft depth 3 — the publisher's operating point
`--max-num-batched-tokens 2048`	prefill chunk	A/B-verified optimum (1024/2048/4096 ladder, see "Tuning levers")
`--prefix-cache-host-mib 4096 --prefix-cache-dir /data/kvcache/frank-primary --prefix-cache-disk-mib 65536`	3 tiers	host tier pinned (see the ledger), disk tier in `/home/dank/radiance-state` (mounted rw at `/data`; `/` is ext4, mapped-tier-eligible); a session snapshot moves with its blocks
`--ngram-placement ram`	PLE pinned to host RAM	Frank's pin law. The PLE n-gram table is 51.2 GB (320,001,536 rows × 160 dims = 29 % of all parameters) served via io_uring reads; `ram` refuses the boot if memlock is too small
`--override-chat-template /templates/template.jinja`	froggeric v22.5	vendored `scripts/qwen38-froggeric-v22.5.jinja` (28,234 B, `qwen3.8-froggeric-v22.5`); rad-convert bakes the checkpoint's own template, this overrides it, and the engine derives the reasoning/tool-call reply format from it

Pinned-RAM ledger (root unit + LimitMEMLOCK=infinity exist for this): 2 × 40 GiB expert pools + 47.7 GiB PLE + 4 GiB prefix host tier + ~2 GiB spare ≈ 134 GiB. ttm.pages_limit (this kernel has NO pages_max — the two-knob story was stale) caps GPU-mappable pinned pages (default = half of RAM): the pre-cutover cap was 123 GiB — BELOW the ledger — so the serve script's TTM preflight (FRANK_PRIMARY_TTM_MIN_MIB=140000) refuses to start until it is raised. DONE 2026-10-05 pre-cutover: sysfs write of 52,428,800 pages = 200 GiB (/sys/module/ttm/parameters/pages_limit; verified read-back; raising at runtime is harmless and allocates nothing). PERSISTENCE: the sysfs write does not survive reboot — add ttm.pages_limit=52428800 to the kernel cmdline at the next scheduled reboot.

Per-rank VRAM at TP2: expert slice 56.4 GiB total across VRAM + host pool; trunk ~3.75 GiB/rank; token embedding vocab-parallel (~1.18 GiB/rank — the bf16 table is 2.43 GiB, sharding hands the other half to KV); lm_head row-sharded; recurrent state 54 MiB/copy/card (2 copies with speculation); KV from the 0.82-side remainder of claimed VRAM at 96 MiB headroom.

The engine contract (all verified in source, not assumed)

/health == /ping: 200 JSON (model_id, in_flight, queue_depth) while serving, 503 only while stopping — the listener binds AFTER the full load, so a bare curl -fsS http://127.0.0.1:7770/health gate is honest (the deployer loops on it; the FreeToken-era "status":"ok" body grep is gone). Boot budget 1800 s (publisher start_period 15 m); measured 2026-10-06: health 200 ~2-3 min after start (~120 s warm) — the big terms are the 125.26 GiB page-in (38.1 s at 3.5 GB/s) and the host-pool pin (PLE 47.68 GiB in 43.9 s).
Sampling (oai.cpp): temperature, top_p, top_k, min_p, typical_p, presence_penalty, frequency_penalty, repetition_penalty, repeat_last_n, chat_template_kwargs, reasoning_effort, enable_thinking, preserve_thinking accepted; unknown params are refused (400) — models.yml's extraBody set is fully inside it; nothing unverified rides the wire. max_tokens is clamped, not refused; seed absent = pick one; response_format supported; logprobs is refused (harness never sends it).
Reasoning: --reasoning-format defaults to auto → thinking in reasoning_content (models.yml reasoningContentField matches). Tool calls: the qwen3.8 dialect → message.tool_calls + finish_reason: tool_calls. No parser flags anywhere — parsers are auto.
Vision: the official checkpoint carries the 27-layer vision tower; the engine runs the Qwen-VL processor (patching, pad-token expansion, 3-part rotary are core). models.yml keeps input: [text, image]; --mm-max-patches available.
/metrics: vLLM-compatible names + radiance:* families — the Dankclaw Prometheus job re-scrapes 192.168.0.235:7770 (plain LAN routing reaches it; the netavark published-port quirk is same-host only). /stats, /sessions, /dashboard for live internals; the repo ships deploy/grafana/radiance.json if a dashboard is wanted.
n > 1 requests: n defaults to max(8, max-num-seqs); stop strings ≤ 64 × 4096 B. model field not enforced.

Deploy + verify

# one-time (already done 2026-10-05): payload fetch + sha256 verify + convert
ssh dankserver 'bash /home/dank/frank-primary-build/build-frank-primary-rad.sh --plan-only'   # declare + plan, no bytes
ssh dankserver 'bash /home/dank/frank-primary-build/build-frank-primary-rad.sh'               # full convert + rad-info
./scripts/deploy.sh dankserver --service frank-primary

Post-cutover checklist (each is observable, not assumed):

Unit: is-active active, is-enabled enabled, container answers /health with 200 JSON.
Boot ledger (engine stdout → /home/dank/logs/frank_primary.log): P2P probe per rank pair, TP=2 shard placement, expert pool coverage line ("backs N % of the worst case … M full-length sessions"), KV pool bytes, PLE pinned-RAM line, drafter depth 3, prefix tiers.
MTP live: a completion's timings object shows draft/accept counts; radiance:*spec* metrics in /metrics.
Froggeric template: a probe prompt renders the v22.5 shape (tool-call + thinking turn both exercise it).
Vision: a ≥256 px solid-color PNG answers the exact color through an image_url content part.
Prometheus target frank-primary green (not never-green).

Cutover measured (2026-10-06, all values observed live)

Container: 1,857,127,997,440 B = 177,109 MiB = 173.0 GiB. Plan-only priced 172.76 GiB — the DATA planes; the written container adds a metadata tail. Floor gates must be set from the real file (the first deploy refused on 178000 > 177109; floor is 177000).
Boot: 125.26 GiB read in 38.1 s (3.5 GB/s), PLE pinned "47.68 GiB in 43.9 s" (the 2 B scale plane with it), each rank's mover holds 25.46 GiB static+experts (19.15 GiB resident experts, 79 %), 16,820 pinned pool slots, health 200 ~2-3 min after start (~120 s warm). No 7-minute vLLM-style boot.
KV ledger: kv_attn 99,956 blocks = 2.48 GiB (399,824 tokens), kv_qsa_bkey 317.24 MiB, kv_gdn_state 4 slots × 54 MiB, PLE conv states < 1 MiB; linear checkpoints 56.37 MiB × 8 slots (3 of 8 card-backed at boot, 5 withdrawn under the 208 MiB floor).
MTP on real traffic (temp 0.7, 220-token ask): draft_n 225 / draft_n_accepted 119 = 52.9 % acceptance, decode 23.8 t/s. ONE sample — the shape's real number needs live-traffic measurement.
Contracts: reasoning_content field (matches models.yml), usage metering separates reasoning_tokens; vision — a 512² solid-red PNG through image_url answers Red (279 prompt tokens; NOTE the 32-token budget is eaten by thinking — give vision probes ≥256 max_tokens); native tool_calls with finish_reason tool_calls.
Boot defects fixed the same day: the SSD prefix tier opened only after the serve script mkdir -p'd the full path (the engine's create is NOT recursive: "cannot create /data/kvcache/..."); the converter's scale-name law, the PLE file-shadow and the symlink-through-raw-index bug are in the payload section above.
Prometheus: /metrics answers 200 with vLLM-compatible families (verified from Dankclaw); the frank-primary job needs a prometheus-service deploy to go green (config pushed, reload deferred behind a live parallel writer's dirty tree).

Tuning levers (in order of likely use)

FRANK_PRIMARY_MAX_SEQS: raise 2 → 3/4 only when the ledger's KV headroom line and real slot occupancy agree; each slot buys a recurrent-state copy (54 MiB/card) and shrinks the per-request KV share.
FRANK_PRIMARY_HOST_POOL_MIB: the ledger's coverage line is the oracle — it prints what the pool backs at the worst case and how many full-length sessions fit. RAM budget: pools + PLE + prefix ≤ ~140 GiB of the host's 246.
FRANK_PRIMARY_EXPERT_VS_CACHE: 0.82 is the measured 99 %-hit point; 0.87 zeroes it. Move only with a measured hit-rate reason.
FRANK_PRIMARY_TP_WIRE: wht6 vs exact — exact trades the +5.6 % prefill for bit-exact reductions; the fused one-shot forms are declared over the seqs × (1+spec) row bound.
FRANK_PRIMARY_EXTRA_ARGS: the boot-ladder escape hatch, expanded unquoted.
rad-tune -m <rad> --tp 2 --num-speculative-tokens 3: benchmarks the tunable space at this model's shapes on this machine, writes $RADIANCE_HOME/tune/<machine>.tune which the engine's selector reads at boot. Declare at the shapes you serve (--tp 2 --num-speculative-tokens 3), or you measure a program the serve never runs. $RADIANCE_HOME is /opt/radiance/share/radiance — in the IMAGE — so a --rm scratch container's cache dies with it: the serve script bind-mounts $FRANK_PRIMARY_STATE_DIR/tune (/home/dank/radiance-state/tune) at /opt/radiance/share/radiance/tune, so the cache (36 GEMM rows, gfx1201-x3-AMD_Radeon_Graphics.tune, generated 2026-10-06 in a stopped-service window) survives and is read at every production boot. The two attention requests have no shape recipe (attn_paged, attn_paged_gate_quant — "see tools/opshapes.h") and are not tuned. RADIANCE_TUNE pins a kernel for an A/B — an instrument, not a setting.
FRANK_PRIMARY_MAX_BATCHED_TOKENS (MNBT): A/B'd 2026-10-06 in a stopped-service window (real-text probes: 20,417-token cold prefill, 1,985-token warm prefill, 256-token decode). The prefill ladder is NON-MONOTONIC with an interior optimum at 2048: 1024 → 2048 → 4096 = 231 → 460 → 448 t/s cold, 221 → 412 → 427 t/s warm. Decode is flat within noise (24.0 / 25.1 / 24.3 t/s); draft acceptance 60.1 / 67.1 / 63.3 %. 2048 wins (peak prefill + peak acceptance) — it stays the production value. 1024 halves prefill (the batch too small to saturate the card); 4096 is a hair off 2048 (the chunk past the kernel's sweet spot).
Diagnostics: rad-info (container: encodings, recipe, profile), --debug-graph (what declared vs what issued), rad-kbench (kernel correctness/speed vs libref), /stats + /sessions (live), and the publisher's scripts/ harnesses (fnpf, fnconc, cstep2, ident, mctools) — note mcserve/fnstop-family scripts pkill the engine binary by name; on this host the service is a systemd unit, so use sudo systemctl stop frank-primary instead.

Rollback

Layer	Path
FreeToken (one card, native FP8 safetensors)	`git checkout frank-primary-freetoken-pr587-20261005 -- scripts/frank-primary.env scripts/frank-primary.service scripts/serve-frank-primary.sh scripts/docker/ tests/frank_primary_unit.bats` + redeploy; image `localhost/frank-primary-freetoken:pr587-77a7201` and payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` stay on disk until the window closes
tcclaviger vLLM fork (TP=2, shape G)	git tag `frank-primary-vllm-shapeG-20261003` (trio + env lore; image still in dank's rootless storage)
4-bit StillDeadcode publisher shape	`qwen3.8-next-flash-fp8-iq4r-moe.rad` + `--host-pool-mib 12288` — the documented fallback if the native container ever proves unbootable

Retire the FreeToken-era orcarouter payload to ~/.AGENTTRASH/ only after the rollback window closes (trash policy, never rm).

20 KiB Raw Blame History Unescape Escape