Frank
2992a7ddd6
- AGENTS.md Model Config: the Protoss names are now the ids on the wire (per-endpoint knob named); verify a rename on /v1/models, never on a chat response — radiance echoes the requested name - OMP Harness.md: harness id == served id; one id per model everywhere - Homepage.md: the docs-page "api id" annotations and the switchover promise are retired - OpenWebUI.md: the id-keyed trio (model row, access_grant, api_configs model_ids) is a foreign-key set — re-key it in the same deploy; plus the contenteditable-composer probe note - frank-secondary/frank-large/frank-primary runbooks: served ids frank-zealot/frank-dragoon/frank-archon, and the bench.conf drop-in that made every frank-primary deploy since 2026-10-08 04:09 restart the staging worktree behind a green health gate
225 lines
13 KiB
Markdown
225 lines
13 KiB
Markdown
# frank-primary — Radiance (StillDeadcode), Swift1.5 Qwen3.8-Flash-Next W4Dn8, 2× R9700 TP=2
|
||
|
||
**Status: LIVE — deployed 2026-10-08 (main `3ca64e1`, e2e 33/33).** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 2× Radeon AI PRO R9700 (gfx1201), TP=2
|
||
|
||
The FP8-native era (`qwen4exp-fp8-native.rad`, 173.0 GiB, no-recipe convert)
|
||
is replaced by a **GPTQ-calibrated 4-bit container** of
|
||
`ukisai/Swift1.5-Qwen3.8-Flash-Next` (BF16, 335.3 GiB, abliterated
|
||
finetune) converted with the recipe
|
||
`scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe` on engine
|
||
**1.0.13** (`stilldeadcode/radiance@sha256:4c3f484564…` — same digest as
|
||
the FP8-native era; 1.2.0 was used ONLY for the calibration serve).
|
||
Same port, alias, container name, systemd unit. Rollback: revert the
|
||
`MODEL_FILE` line in `scripts/frank-primary.env` + redeploy —
|
||
`/home/dank/models/rad/qwen4exp-fp8-native.rad` stays on disk (trash
|
||
policy; 173 GiB, purge when the window closes).
|
||
|
||
## Why this swap
|
||
|
||
- **Context economics at production memory budgets**: the FP8-native
|
||
experts (56.4 GiB/rank slice) forced a 40 GiB/rank pinned host pool and
|
||
left KV ~3.5 GiB. W4Dn8 experts cut the resident set to ~1/4, freeing
|
||
host and VRAM budget for KV and slots at equal-or-better measured
|
||
decode (gate: ≥ 30.8 t/s vs the 30.08 t/s control).
|
||
- **Swift 1.5 base**: abliterated finetune of Flash-Next (Frank's
|
||
fleet-personality call; frank-large serves the same family).
|
||
- **GPTQ closes the precision gap**: RTN alone on BF16 weights lands
|
||
measurably worse; GPTQ with Hessians from a 10M-token corpus of this
|
||
model's own distribution recovers most of it, verified end-to-end by a
|
||
KLD gate against the BF16 container itself (KL ≤ 0.03, top-1 ≥ 95 %).
|
||
|
||
## The payload chain
|
||
|
||
1. **Fetch**: `scripts/download-frank-primary-swift15.sh` →
|
||
`/home/dank/models/Swift1.5-Qwen3.8-Flash-Next` (131 shards, 335.3
|
||
GiB, per-shard LFS sha256 verified, no token on the command line).
|
||
48 layers, 512 experts (top-10 routed), moe_inter 640, hidden 2560,
|
||
PLE 20M-vocab table 95.37 GiB, MTP head, vision tower.
|
||
2. **BF16 container**: `run-swift15-bf16.sh convert` →
|
||
`rad-convert` no-recipe (pure copy) at `--max-ctx 4096` →
|
||
`qwen4exp-swift15-bf16.rad` (335.28 GiB). Serves on 1.0.13 as the
|
||
KLD reference engine (disk-tier placement; boot ~35 min).
|
||
3. **KLD reference**: `run-swift15-bf16.sh kld` (1.0.13) →
|
||
`/home/dank/corpora/kld-ref` — `radiance-kld-1` format, `rows:
|
||
50235`, 197 source-labelled docs, top-16 log-probs (f16), corpus
|
||
fnv1a64 `a9d62b33c18b3d9c`. ~4 h at 32k prefill steps
|
||
(`--max-num-batched-tokens 32768` — a 65536 step OOMs an arena
|
||
tensor at `--max-model-len 16384`). **The wrapper's exit line is the
|
||
receipt: never scp this script while it runs** (bash lazy-read;
|
||
a mid-run swap killed a clean engine exit with rc=127, third fleet
|
||
occurrence of the script-swap law). `kld-then-calib.sh SKIP_KLD=1`
|
||
resumes from the `rows: 50235` receipt.
|
||
4. **Calibration**: `run-swift15-bf16.sh calib` on **1.2.0** (the only
|
||
engine with `RADIANCE_CALIB_TOKENS`; grams are engine-invariant
|
||
bf16 activations — RADGRAM2 format byte-identical across versions)
|
||
+ `calib-push.py` pushing `calib-corpus.jsonl` (10,014,932 tokens,
|
||
1,294 prompts, assembled from real HF sources, chat-templated,
|
||
source-labelled). The tap (in the `qwen4exp_fp8.so` architecture
|
||
plugin) writes per-expert down-proj Gram matrices
|
||
`gram.r<rank>.blk.L.ffn_down_exps.<e>.bin` (32-byte RADGRAM2 header +
|
||
640×640 f32; `rows` = routing count, `trace` = activation energy)
|
||
and stops the serve at the token threshold. TP=2: whole experts per
|
||
rank (256/rank); gate_up input is rank-sliced (plugin warning — the
|
||
recipe only casts the down projection).
|
||
5. **Protected-expert derivation**: `derive-protected-experts.py
|
||
/home/dank/corpora/calib` → the 24 highest-energy experts (the law
|
||
is data-driven; Swift drew 24, not the fp8-era 10 — its routing is
|
||
spikier: `blk.47 e445` alone carries 32 % of its layer's down energy
|
||
at 70× routing share) promoted to whole-expert bf16.
|
||
Protection law: energy share ≥ 3 % AND ≥ 3× routing share; even count
|
||
per rank half; odd half gains the next most energetic.
|
||
`apply-protected-cast.sh` splices the derived names between the
|
||
`# PROTECTED-BEGIN/END` markers of the recipe (refuses double-derive).
|
||
6. **Final convert**: `convert-swift15-w4dn8.sh` (1.0.13, host-backend
|
||
CPU-only — no GPU visible to `rad-convert`; GPTQ solve is CPU BLAS) —
|
||
gates: ≥48 gram files present, bf16 container already deleted
|
||
(disk), ≥146 GiB free, output ≥108,000 MiB, `rad-info` shows the
|
||
gptq receipt. No `--reuse`. Measured: 5 h 50 m (6.0–6.5 min/layer on
|
||
49 GPTQ layers, load-average 8), output **116,416 MiB**.
|
||
The plugin's `rad_arch_declare` enforces ONE ENCODING PER EXPERT
|
||
TABLE (`one table of experts takes one encoding, and only a whole
|
||
expert kept plain bf16 is served beside it`) — a split tier
|
||
(down=fp8, gate_up=w4) is unservable: the kernel serves a single
|
||
w4nl64a8h table; both expert tables ride GPTQ w4 (down calibrated,
|
||
gate_up uncalibrated), protected experts bf16 beside them.
|
||
|
||
## The recipe tiers (scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe)
|
||
|
||
| Tier | Encoding | Why |
|
||
|---|---|---|
|
||
| MoE experts, both `gate_up` and `down` (one w4 table/layer) | **GPTQ w4nl64a8h** — u4 codes, fwht128 rotation, fp8 scales/64 + f32 scale plane | the campaign tier: GPTQ on down-input Hessians from the 10M-token corpus (the tap rides the down projection; gate_up shares the table uncalibrated — one-encoding-per-table plugin law) |
|
||
| 24 derived protected experts (whole) | **bf16** | energy-share law (step 5) |
|
||
| shared expert + trunk attention/SSM linears | **int8**, scale per 128 cols, searched | trunk precision law |
|
||
| hyper-cond + MTP head | **fp8_e4m3 row-groups** | hc+mtp fp8 law |
|
||
| lm_head | **int8** | i8 lm_head law (E4M3 costs 0.6 top-1 alone) |
|
||
| PLE n-gram table | fp8 at the checkpoint's fixed scale 1.9931793212890625e-04 | the fixed-scale law; absmax audit passed the 2⁻¹³..2⁻¹⁰ plateau |
|
||
| embeddings, vision tower, QSA indexer | **bf16 passthrough** | untouched |
|
||
|
||
## The shape (`scripts/frank-primary.env`)
|
||
|
||
Unchanged from the FP8-native era: TP=2 + `wht6`, `--max-num-seqs 3`,
|
||
`--max-model-len 200000`, MNBT 2048 (the MTP resonance law — 2560
|
||
collapses decode to 2.6–7.7 t/s), fp8 KV, `expert_tiered` host pool
|
||
40960 (the load-bearing floor), `--ngram-placement ram`, froggeric
|
||
v22.5 template override. `MODEL_FILE` points at
|
||
`/home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416
|
||
MiB measured**); the serve size floor `FRANK_PRIMARY_RAD_MIN_MIB` moves
|
||
177000 → **110000**.
|
||
|
||
## Gates (measured at cutover)
|
||
|
||
- **KLD gate: PASS (2026-10-08 04:12)** — `kld-gate-swift15.sh`,
|
||
candidate on 1.0.13 vs `kld-ref`: **mean KL 0.0146** (median 0.00126,
|
||
p90 0.0315, p99 0.194, max 4.12), **top-1 97.84 %**,
|
||
**ppl ratio 1.0126** (candidate 1.6913 vs reference 1.6702), 50,235
|
||
positions / 197 docs. Report: `/home/dank/radiance-state/kld-gate.json`
|
||
(+ `.rows` per position). Gate shape: disk-tier, pool 40960, MNBT 3072
|
||
(the w4 activation arena scales hard with MNBT: 32768 → 33.3 GiB arena,
|
||
refused), KV 2048, headroom 512, no speculation. The engine ran rank 0
|
||
fully on card 0 (25.88 GiB) with rank 1 host-side in this shape —
|
||
scoring is placement-invariant; scored at 28 positions/s.
|
||
- **KLD gate re-PASS (2026-10-08 04:49) at the production-legal shape**:
|
||
MNBT 2048 (the w4 file-tier law: experts stream in place only at
|
||
MNBT ≤ 2048 — the recording's 32768 demands a widened form the disk
|
||
tier refuses), pool 40960, vram-kv 3072 fp8, tp 2, headroom 768.
|
||
**mean KL 0.0131, top-1 97.92 %**. The Δ vs the first PASS (0.0015) is
|
||
the float-order noise band of the chunk-size change — the verdict is
|
||
chunk-size invariant, ~2.3× inside the 0.03 bar on both shapes.
|
||
- **Throughput gate: PASS (2026-10-08 05:40)** — decode at 44.5k depth
|
||
(streamed chat probe, temp 0, 512 tokens, salted): arm A (main serve
|
||
shape, EVC 0.82) **146–173 t/s**, engine timings **121.9 t/s**,
|
||
draft accept **0.7246** (350/483); arm C (deployed vram-pair shape
|
||
26071/3543) median **136.0 t/s**, warm TTFT 9.45 s. Control
|
||
(fp8-native 1.0.13): 30.08 / accept 0.7549. Prefill ~4.7k t/s warm.
|
||
The w4 expert table is resident — the fp8-native era's NVMe expert
|
||
streaming (the 3.5 GB/s wall) is gone; the MTP ×3 head rides a 45–60
|
||
t/s base decode. Conversation battery (short context): geomean
|
||
172 t/s (min 100 / max 208), mean ttft 0.09 s. The EVC ladder is moot:
|
||
the w4 resident set fits fully inside the 40 GiB pool at 0.82
|
||
(ledger: mover full fill). Depth-output audit: the 44.5k-depth
|
||
summary is a coherent ordered index of the concatenated corpus
|
||
(no repetition, no NaN-token garbage) — the depth-corruption class
|
||
of 2026-10-04 is clean.
|
||
- Fleet probes: PASS, re-run on the deployed prod — vision (solid-red
|
||
512 px → "Red"), tool_calls (`{"expr":"23 * 47"}`), reasoning_effort
|
||
xhigh/medium/low/high all 200 (the Swift1.5 froggeric template accepts
|
||
the full set), reasoning_content present, response_format json_object
|
||
honoured. **Vision probe law:** budget ≥ 256 tokens and read
|
||
`reasoning_content` as fallback — at 64 tokens thinking exhausts the
|
||
budget and `content` comes back empty (the answer lands in the
|
||
reasoning channel).
|
||
- **Deployed verification (2026-10-08)**: `deploy.sh dankserver
|
||
--service frank-primary` green incl. Playwright e2e 33/33; unit
|
||
`active`, `/health` 200, `/v1/models` id `frank-primary` (the harness
|
||
alias is the served-model-name, independent of the file — models.yml
|
||
untouched). Deployed shape: fp8 KV (KLD-gate shape), spec depth 2,
|
||
seqs 3, vram pair 26071/3543. Boot ledger: KV pool 3.46 GiB, expert
|
||
pools 57.84 GiB, cards settled 1.41 GiB free, `serving on :7770`.
|
||
Live decode probe (512 tokens, streamed, temp 0): ~190 t/s.
|
||
|
||
### Wire rename + the bench drop-in trap (2026-10-09)
|
||
|
||
`FRANK_PRIMARY_SERVED_NAME` moved `frank-primary` → **frank-archon**; the
|
||
service, unit, scripts and env file keep the old name. Radiance is the only
|
||
fleet engine whose `/v1/models` listing reflects the flag, and it echoes the
|
||
**requested** model name in chat responses — so a chat probe that returns 200
|
||
proves nothing about the served id. Always read `/v1/models`.
|
||
|
||
The first deploy after the edit came up green and `/v1/models` still said
|
||
`frank-primary`. Cause: a campaign leftover drop-in
|
||
`/etc/systemd/system/frank-primary.service.d/bench.conf` (2026-10-08 04:09)
|
||
overrode BOTH `Environment=FRANK_PRIMARY_ENV_FILE` and `ExecStart` to the
|
||
staging worktree `/home/dank/serverAppConfigs-swift15`. Every
|
||
`deploy.sh dankserver --service frank-primary` since then restarted the
|
||
staging tree: green health gate, old configuration, silently. Verified before
|
||
removing: the staging `serve-frank-primary.sh` and `frank-primary.service`
|
||
are byte-identical to the checkout, its `frank-primary.env` differed only by
|
||
the alias line, and its HEAD (`8375567`) is an ancestor of `main`. The
|
||
drop-in is staged in `/home/dank/.AGENTTRASH/20261009-frank-primary-bench-dropin/`.
|
||
After `daemon-reload` + restart, `/v1/models` reports `frank-archon` (boot
|
||
healthy in 102 s, shape unchanged: fp8 KV, spec depth 2, seqs 3, vram pair
|
||
26071/3543).
|
||
|
||
**Deployer blind spot:** `deploy.sh` plans from repo paths and gates on the
|
||
port. It cannot see host-side drop-ins, so unit drift is invisible to a green
|
||
run. After an LLM deploy, verify the served configuration behaviourally
|
||
(here: `podman inspect <name> --format {{json .Config.Cmd}}` and `/v1/models`).
|
||
|
||
|
||
|
||
## Deploy + verify
|
||
|
||
```bash
|
||
sudo systemctl stop frank-primary # drained window
|
||
bash convert-swift15-w4dn8.sh # if not already run
|
||
./scripts/deploy.sh dankserver --service frank-primary
|
||
```
|
||
|
||
Post-cutover checklist: unit active/enabled; `/health` 200; boot ledger
|
||
(expert pool coverage now w4-sized, KV pool should be larger than the
|
||
FP8-native era's 399,824 tokens); MTP draft counts in `timings`; froggeric
|
||
probe; ≥256 px `image_url` vision probe answers the color; tool-call
|
||
probe; `reasoning_effort` fold; Prometheus target green; `tests/run.sh`
|
||
+ Playwright e2e green.
|
||
|
||
## Rollback
|
||
|
||
| Layer | Path |
|
||
|---|---|
|
||
| FP8-native radiance (immediate) | revert `MODEL_FILE` in `scripts/frank-primary.env` + redeploy; `qwen4exp-fp8-native.rad` (173.0 GiB) on disk |
|
||
| FreeToken PR #587 | git tag `frank-primary-freetoken-pr587-20261005` |
|
||
| tcclaviger vLLM shape G | git tag `frank-primary-vllm-shapeG-20261003` |
|
||
|
||
Campaign artifacts on disk: BF16 checkpoint (335 GiB), `kld-ref` (24
|
||
GiB), grams (`/home/dank/corpora/calib`), corpora JSONL, logs
|
||
`/home/dank/logs/swift15_*.log`. The BF16 `.rad` container is deleted
|
||
before the final convert (disk gate) — the raw checkpoint is the
|
||
re-convert source if the window ever reopens.
|
||
|
||
## Links
|
||
|
||
- [[frank-primary-radiance-qwen38-native-fp8]] — the replaced FP8-native era
|
||
- [[LLM/index]] — fleet index
|
||
- [[swift15-campaign-worklog]] — campaign record
|
||
- [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here
|