frank-primary: radiance native-FP8 TP=2 cutover (engine, payload build, rollback)
Replace the FreeToken PR587 endpoint (single-GPU: its TP=2 dies in the pynccl.cu ROCm JIT) with the radiance engine on both R9700s, TP=2 via libr4d's fused one-shot all-reduce. Official Qwen/Qwen3.8-Flash-Next-FP8 payload at native block-FP8 precision: - scripts/build-frank-primary-rad.sh: staging (quantization_config drop = plugin SELECTION law) -> checkpoint reshape -> rad-convert with ONE --quant rule (PLE rtn fixed-scale); plan-only first. - scripts/stack-qwen38-official-fp8.py: the two value-preserving reshapes (experts byte-stacked into the plugin's stacked layout; PLE's 128 scaleless fp8 shards decoded to one bf16 shard with the checkpoint's own scale; the rtn rule re-encodes at that same scale - codes round-trip exactly and land the one-element scale plane the plugin declares). - scripts/hfget.sh + verified download (131/131 shards sha256-match). - froggeric v22.5 chat template vendored (--override-chat-template). - serve script (pinned ledger: 2x40 GiB pools + 47.7 GiB PLE + prefix host), TTM preflight fixed for kernels with pages_limit only (no pages_max - an absent knob is skipped, not read as zero); pages_limit raised to 52,428,800 pages = 200 GiB on the host pre-cutover. - root unit, relinked; deployer gates image digest + .rad floor 158,000 MiB + health; prometheus re-adds the /metrics job. - rollback: tag frank-primary-freetoken-pr587-20261005 (FreeToken trio + Dockerfile); payload + image stay on disk until the window closes. deploy-remote.sh also carries the parallel session's frank-small payload gates (swift15) - committed here per Frank's direction to commit the tree.
This commit is contained in:
Frank
@@ -1,6 +1,10 @@
|
||||
# frank-primary — FreeToken PR #587 ROCm (Qwen3.8-Flash-Next-Uncensored-FP8, 1× R9700)
|
||||
|
||||
**Status: ACTIVE 2026-10-05 →** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 1× Radeon AI PRO R9700 (gfx1201) — the second R9700 sits idle
|
||||
**Status: RETIRED 2026-10-05 → (superseded the same day by the radiance
|
||||
cutover — see [[frank-primary-radiance-qwen38-native-fp8]]; kept as the
|
||||
rollback runbook: the image + orcarouter payload stay on disk until the
|
||||
window closes)** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 ·
|
||||
**GPU:** 1× Radeon AI PRO R9700 (gfx1201) — the second R9700 sits idle
|
||||
|
||||
The tcclaviger RDNA4 vLLM fork was replaced by **FreeToken at PR #587 head
|
||||
`77a7201f98f00d93f618c6cf8b81ffe8360e0899`** (branch `rocm714_torch214`), built
|
||||
|
||||
@@ -0,0 +1,224 @@
|
||||
# frank-primary — Radiance (StillDeadcode), Qwen3.8-Flash-Next-FP8 NATIVE, 2× R9700 TP=2
|
||||
|
||||
**Status: ACTIVE 2026-10-05 →** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 2× Radeon AI PRO R9700 (gfx1201), TP=2
|
||||
|
||||
The FreeToken PR #587 endpoint was replaced by the **radiance engine** (C++/HIP,
|
||||
libr4d kernels for gfx1201, no Python in the serving path) driving the
|
||||
digest-pinned image `stilldeadcode/radiance@sha256:4c3f484564bb2872784e4…`
|
||||
(v1.0.7 line, 287 MiB, `libr4d@0.5.0`) as a plain `podman run` through
|
||||
`scripts/serve-frank-primary.sh` under the **root systemd unit**
|
||||
(`frank-primary.service`, `User=dank`, `LimitMEMLOCK=infinity`). Same port,
|
||||
same served alias (`frank-primary`), same container identity
|
||||
(`frank-primary` = the unit name). Rollback: git tag
|
||||
`frank-primary-freetoken-pr587-20261005` (FreeToken trio + Dockerfile), image
|
||||
`localhost/frank-primary-freetoken:pr587-77a7201`, payload
|
||||
`/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8`.
|
||||
|
||||
## Why radiance
|
||||
|
||||
- **TP=2 is real here.** FreeToken's TP=2 dies in the `pynccl.cu` ROCm JIT
|
||||
(`cuda_runtime.h` unresolvable for gfx1201 — probed 2026-10-06); the
|
||||
second R9700 idled. Radiance's TP wire is libr4d's own one-shot
|
||||
all-reduce (no RCCL), legal on this board since the 2026-09-20 swap
|
||||
(ProArt B850, x8/x8 on CPU root ports, `Routing+`/`ReqEn+` everywhere).
|
||||
- **Native FP8 end to end** (Frank's call): the official
|
||||
`Qwen/Qwen3.8-Flash-Next-FP8` checkpoint keeps its modelopt block-FP8
|
||||
encoding — no requantisation, no calibration run, no 4-bit codebook.
|
||||
- **Every tier is explicit and measured**, not an estimator: placement
|
||||
`expert_tiered` with a pinned host pool, `--ngram-placement ram` for the
|
||||
47.7 GiB PLE table, `--expert-vs-cache-ratio 0.82` (the publisher's
|
||||
measured 99 %-hit point; 0.87 already collapses to 0 %), fp8 KV.
|
||||
|
||||
## The payload: native FP8, reshaped then converted (one recipe rule)
|
||||
|
||||
- **Raw checkpoint**: `/home/dank/models/Qwen/Qwen3.8-Flash-Next-FP8`
|
||||
(ungated repo, 172.8 GB, 131 shards + index + tokenizer + vision
|
||||
configs). Fetched by `scripts/download-frank-primary.sh` (vendored
|
||||
`hfget.sh`, curl+jq, no HF token needed) and **verified byte-for-byte**:
|
||||
all 131 shards sha256-match the Hub LFS oids (2026-10-05, log
|
||||
`/home/dank/logs/verify-qwen38-official-fp8.log`).
|
||||
- **The container**: `/home/dank/models/rad/qwen4exp-fp8-native.rad`
|
||||
(~161 GiB; floor `FRANK_PRIMARY_RAD_MIN_MIB=158000` in the env gates a
|
||||
truncated convert). Built by `scripts/build-frank-primary-rad.sh`:
|
||||
1. stage `/home/dank/models/stage-qwen4exp-fp8` — `config.json` with
|
||||
**`quantization_config` dropped** + relative symlinks into the raw
|
||||
checkout;
|
||||
2. `stack-qwen38-official-fp8.py` reshapes the staging dir — two
|
||||
transforms, both value-preserving (script docstring carries the full
|
||||
argument): the plugin maps the STACKED expert layout
|
||||
(`mlp.experts.gate_up_proj` `[512,1280,2560]` + `_scale_inv` at the
|
||||
same index) while the official repo ships per-expert unfused tensors,
|
||||
so staging byte-stacks all 49 stacks' (48 trunk + MTP) 512 experts
|
||||
with NO requantisation; and the PLE table ships as 128 **scaleless**
|
||||
E4M3 shards + one bf16 scalar (1.9931793212890625e-04) while the
|
||||
plugin's fp8 branch wants the codes beside a ONE-ELEMENT bf16 scale
|
||||
plane — so staging decodes to a single bf16 `shard_0` with the
|
||||
checkpoint's own scale (e4m3 decode and the scale are both exact in
|
||||
bf16);
|
||||
3. `rad-convert` with ONE `--quant` rule — pure copy for everything else
|
||||
(every weight keeps the checkpoint's block-FP8 planes: `X.weight`
|
||||
E4M3 codes + `X.weight_scale_inv` bf16 `[128×128]` scales) — and the
|
||||
rtn rule re-encodes the bf16 PLE table at the same fixed scale, an
|
||||
exact round-trip (nearest-code distance 2⁻³ vs the bf16 rounding
|
||||
perturbation 2⁻⁹): the container gets the checkpoint's own PLE codes
|
||||
plus the one-element scale plane (`blk.1.ple_ngram.scale`, view {1});
|
||||
4. `rad-info` summary + size floor.
|
||||
- **Why the staging dir is mandatory** (the descriptor law, spec §2.4):
|
||||
the plugin is selected by the exact `(arch_id, quant descriptor)` pair
|
||||
with no fallback. A raw official checkpoint carries
|
||||
`quantization_config.quant_method: "fp8"` → descriptor `"fp8"` → **no
|
||||
shipped plugin claims `(qwen4exp, "fp8")`** — raw serve AND unmodified
|
||||
convert both refuse. The shipped `qwen4exp_fp8` plugin claims
|
||||
`(qwen4exp, "")` (recipe-built containers). `rad_convert.cpp` writes
|
||||
`w.set_quant(meta.quant)` into the header, so the staged config's empty
|
||||
quant method flows through as the container descriptor while **every
|
||||
weight stays FP8 data**. Native precision is untouched; only the
|
||||
selection key differs.
|
||||
- `--plan-only` first: the convert's declare phase is the engine's own,
|
||||
so a plan that resolves every op will boot. The convert runs card-less
|
||||
(declare = CPU-side layout queries); add `--device` only if a plugin
|
||||
load ever demands it.
|
||||
|
||||
## The shape (`scripts/frank-primary.env`, every value commented there)
|
||||
|
||||
| Knob | Value | Why |
|
||||
|---|---|---|
|
||||
| Image | `stilldeadcode/radiance@sha256:4c3f4845…` | digest-pinned INDEX digest (RepoDigests also carry the per-arch manifest `1c582005…`; the image ID `69d05724…` is the config digest, NOT a manifest digest — pinning it sends `podman run` to the network); entrypoint = engine, tools (`rad-convert`/`rad-info`/`rad-tune`/`rad-kbench`) on PATH |
|
||||
| Container | `frank-primary` | identity = service name across runtimes; legacy list retires `frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe` |
|
||||
| `--tp 2 --tp-wire wht6` | both R9700s | fused WHT-6 lossy wire: +5.6–6.3 % prefill, ~−1.3 % 8-stream decode at TP2; `--tp-wire-min-kb` stays engine default (128 KiB) |
|
||||
| `--max-num-seqs 2` | first boot | Frank: "parallelism = 2 to start". Raise only after a real-traffic look at slot occupancy + the ledger's KV headroom line. The fused all-reduce row count is `seqs × (1 + spec) = 8` |
|
||||
| `--max-model-len 200000` | served window | native 262,144; the publisher's production value and what `models.yml` quotes. Longer prompt = refused, never truncated |
|
||||
| `--placement expert_tiered --host-pool-mib 40960` | pinned expert pool **per rank** | native FP8 experts are 112.8 GiB total (56.4 GiB/rank slice) vs ~29 GiB at the 4-bit publisher shape — the pool is the price of native precision. Resize from the boot ledger's coverage line, never arithmetic |
|
||||
| `--gpu-headroom-mib 96 --expert-vs-cache-ratio 0.82` | publisher values | 0.82 = measured 99 % expert-cache hit; 0.87 → 0 % (the cliff) |
|
||||
| `--kv-cache-dtype fp8` | **7.32 KiB/token** | 12 QSA full-attention layers, 2 KV heads × 256 dim, K+V; a 200K-token sequence is 1.43 GiB. KV is sized from the VRAM the ratio leaves |
|
||||
| `--num-speculative-tokens 3` | MTP head | the checkpoint carries `mtp.*` (3,101 tensors, `mtp_num_hidden_layers: 1`); the plugin's probe returns draft depth 3 — the publisher's operating point |
|
||||
| `--max-num-batched-tokens 2048` | prefill chunk | matches the `--checkpoint-interval` default |
|
||||
| `--prefix-cache-host-mib 4096 --prefix-cache-dir /data/kvcache/frank-primary --prefix-cache-disk-mib 65536` | 3 tiers | host tier pinned (see the ledger), disk tier in `/home/dank/radiance-state` (mounted rw at `/data`; `/` is ext4, mapped-tier-eligible); a session snapshot moves with its blocks |
|
||||
| `--ngram-placement ram` | **PLE pinned to host RAM** | Frank's pin law. The PLE n-gram table is 51.2 GB (320,001,536 rows × 160 dims = 29 % of all parameters) served via io_uring reads; `ram` **refuses the boot** if memlock is too small |
|
||||
| `--override-chat-template /templates/template.jinja` | froggeric v22.5 | vendored `scripts/qwen38-froggeric-v22.5.jinja` (28,234 B, `qwen3.8-froggeric-v22.5`); rad-convert bakes the checkpoint's own template, this overrides it, and the engine derives the reasoning/tool-call reply format from it |
|
||||
|
||||
**Pinned-RAM ledger** (root unit + `LimitMEMLOCK=infinity` exist for this):
|
||||
2 × 40 GiB expert pools + 47.7 GiB PLE + 4 GiB prefix host tier + ~2 GiB
|
||||
spare ≈ **134 GiB**. `ttm.pages_limit` (this kernel has NO `pages_max` —
|
||||
the two-knob story was stale) caps GPU-mappable pinned pages (default = half
|
||||
of RAM): the pre-cutover cap was **123 GiB** — BELOW the ledger — so the
|
||||
serve script's TTM preflight (`FRANK_PRIMARY_TTM_MIN_MIB=140000`) refuses to
|
||||
start until it is raised. DONE 2026-10-05 pre-cutover: sysfs write of
|
||||
**52,428,800 pages = 200 GiB** (`/sys/module/ttm/parameters/pages_limit`;
|
||||
verified read-back; raising at runtime is harmless and allocates nothing).
|
||||
PERSISTENCE: the sysfs write does not survive reboot — add
|
||||
`ttm.pages_limit=52428800` to the kernel cmdline at the next scheduled
|
||||
reboot.
|
||||
|
||||
**Per-rank VRAM** at TP2: expert slice 56.4 GiB total across VRAM + host
|
||||
pool; trunk ~3.75 GiB/rank; token embedding vocab-parallel (~1.18 GiB/rank
|
||||
— the bf16 table is 2.43 GiB, sharding hands the other half to KV); lm_head
|
||||
row-sharded; recurrent state 54 MiB/copy/card (2 copies with speculation);
|
||||
KV from the `0.82`-side remainder of claimed VRAM at 96 MiB headroom.
|
||||
|
||||
## The engine contract (all verified in source, not assumed)
|
||||
|
||||
- **`/health` == `/ping`**: 200 JSON (`model_id, in_flight, queue_depth`)
|
||||
while serving, 503 only while stopping — **the listener binds AFTER the
|
||||
full load**, so a bare `curl -fsS http://127.0.0.1:7770/health` gate is
|
||||
honest (the deployer loops on it; the FreeToken-era `"status":"ok"` body
|
||||
grep is gone). Boot budget 1800 s (publisher `start_period` 15 m); the
|
||||
big terms are the ~161 GiB page-in and the host-pool pin.
|
||||
- **Sampling** (oai.cpp): `temperature, top_p, top_k, min_p, typical_p,
|
||||
presence_penalty, frequency_penalty, repetition_penalty, repeat_last_n,
|
||||
chat_template_kwargs, reasoning_effort, enable_thinking,
|
||||
preserve_thinking` accepted; **unknown params are refused (400)** —
|
||||
models.yml's extraBody set is fully inside it; nothing unverified rides
|
||||
the wire. `max_tokens` is clamped, not refused; `seed` absent = pick
|
||||
one; `response_format` supported; **`logprobs` is refused** (harness
|
||||
never sends it).
|
||||
- **Reasoning**: `--reasoning-format` defaults to `auto` → thinking in
|
||||
`reasoning_content` (models.yml `reasoningContentField` matches). Tool
|
||||
calls: the qwen3.8 dialect → `message.tool_calls` +
|
||||
`finish_reason: tool_calls`. No parser flags anywhere — parsers are
|
||||
auto.
|
||||
- **Vision**: the official checkpoint carries the 27-layer vision tower;
|
||||
the engine runs the Qwen-VL processor (patching, pad-token expansion,
|
||||
3-part rotary are core). `models.yml` keeps `input: [text, image]`;
|
||||
`--mm-max-patches` available.
|
||||
- **`/metrics`**: vLLM-compatible names + `radiance:*` families — the
|
||||
Dankclaw Prometheus job re-scrapes 192.168.0.235:7770 (plain LAN routing
|
||||
reaches it; the netavark published-port quirk is same-host only).
|
||||
`/stats`, `/sessions`, `/dashboard` for live internals; the repo ships
|
||||
`deploy/grafana/radiance.json` if a dashboard is wanted.
|
||||
- **`n` > 1 requests**: `n` defaults to `max(8, max-num-seqs)`; stop
|
||||
strings ≤ 64 × 4096 B. `model` field not enforced.
|
||||
|
||||
## Deploy + verify
|
||||
|
||||
```bash
|
||||
# one-time (already done 2026-10-05): payload fetch + sha256 verify + convert
|
||||
ssh dankserver 'bash /home/dank/frank-primary-build/build-frank-primary-rad.sh --plan-only' # declare + plan, no bytes
|
||||
ssh dankserver 'bash /home/dank/frank-primary-build/build-frank-primary-rad.sh' # full convert + rad-info
|
||||
./scripts/deploy.sh dankserver --service frank-primary
|
||||
```
|
||||
|
||||
Post-cutover checklist (each is observable, not assumed):
|
||||
|
||||
1. Unit: `is-active` active, `is-enabled` enabled, container answers
|
||||
`/health` with 200 JSON.
|
||||
2. Boot ledger (engine stdout → `/home/dank/logs/frank_primary.log`):
|
||||
P2P probe per rank pair, TP=2 shard placement, expert pool coverage
|
||||
line ("backs N % of the worst case … M full-length sessions"), KV pool
|
||||
bytes, PLE pinned-RAM line, drafter depth 3, prefix tiers.
|
||||
3. MTP live: a completion's `timings` object shows draft/accept counts;
|
||||
`radiance:*spec*` metrics in `/metrics`.
|
||||
4. Froggeric template: a probe prompt renders the v22.5 shape
|
||||
(tool-call + thinking turn both exercise it).
|
||||
5. Vision: a ≥256 px solid-color PNG answers the exact color through an
|
||||
`image_url` content part.
|
||||
6. Prometheus target `frank-primary` green (not never-green).
|
||||
|
||||
## Tuning levers (in order of likely use)
|
||||
|
||||
- **`FRANK_PRIMARY_MAX_SEQS`**: raise 2 → 3/4 only when the ledger's KV
|
||||
headroom line and real slot occupancy agree; each slot buys a
|
||||
recurrent-state copy (54 MiB/card) and shrinks the per-request KV share.
|
||||
- **`FRANK_PRIMARY_HOST_POOL_MIB`**: the ledger's coverage line is the
|
||||
oracle — it prints what the pool backs at the worst case and how many
|
||||
full-length sessions fit. RAM budget: pools + PLE + prefix ≤ ~140 GiB of
|
||||
the host's 246.
|
||||
- **`FRANK_PRIMARY_EXPERT_VS_CACHE`**: 0.82 is the measured 99 %-hit
|
||||
point; 0.87 zeroes it. Move only with a measured hit-rate reason.
|
||||
- **`FRANK_PRIMARY_TP_WIRE`**: `wht6` vs `exact` — `exact` trades the
|
||||
+5.6 % prefill for bit-exact reductions; the fused one-shot forms are
|
||||
declared over the seqs × (1+spec) row bound.
|
||||
- **`FRANK_PRIMARY_EXTRA_ARGS`**: the boot-ladder escape hatch, expanded
|
||||
unquoted.
|
||||
- **`rad-tune -m <rad> --tp 2`**: benchmarks the tunable space at this
|
||||
model's shapes on this machine, writes `$RADIANCE_HOME/tune/<machine>.tune`
|
||||
which the engine's selector reads at boot. Declare at the shapes you
|
||||
serve (`--tp 2`), or you measure a program the serve never runs.
|
||||
`RADIANCE_TUNE` pins a kernel for an A/B — an instrument, not a setting.
|
||||
- **Diagnostics**: `rad-info` (container: encodings, recipe, profile),
|
||||
`--debug-graph` (what declared vs what issued), `rad-kbench` (kernel
|
||||
correctness/speed vs libref), `/stats` + `/sessions` (live), and the
|
||||
publisher's `scripts/` harnesses (`fnpf`, `fnconc`, `cstep2`, `ident`,
|
||||
`mctools`) — note `mcserve/fnstop`-family scripts `pkill` the engine
|
||||
binary by name; on this host the service is a systemd unit, so use
|
||||
`sudo systemctl stop frank-primary` instead.
|
||||
|
||||
## Rollback
|
||||
|
||||
| Layer | Path |
|
||||
|---|---|
|
||||
| FreeToken (one card, native FP8 safetensors) | `git checkout frank-primary-freetoken-pr587-20261005 -- scripts/frank-primary.env scripts/frank-primary.service scripts/serve-frank-primary.sh scripts/docker/ tests/frank_primary_unit.bats` + redeploy; image `localhost/frank-primary-freetoken:pr587-77a7201` and payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` stay on disk until the window closes |
|
||||
| tcclaviger vLLM fork (TP=2, shape G) | git tag `frank-primary-vllm-shapeG-20261003` (trio + env lore; image still in dank's rootless storage) |
|
||||
| 4-bit StillDeadcode publisher shape | `qwen3.8-next-flash-fp8-iq4r-moe.rad` + `--host-pool-mib 12288` — the documented fallback if the native container ever proves unbootable |
|
||||
|
||||
Retire the FreeToken-era orcarouter payload to `~/.AGENTTRASH/` only after
|
||||
the rollback window closes (trash policy, never `rm`).
|
||||
|
||||
## Links
|
||||
|
||||
- [[LLM/index]] — fleet index
|
||||
- [[frank-primary-freetoken-pr587]] — the retired FreeToken era
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the retired tcclaviger vLLM era
|
||||
- [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here
|
||||
- [[RDNA4 Container Serving]] — rootless-podman GPU laws on these hosts
|
||||
- `serverAppConfigs-docs/Architecture/Testing Policy.md` — the seam contract
|
||||
Reference in New Issue
Block a user