The endpoint moved to radiance 1.8.1 with MAX_SEQS=2 and the --vram-weights-mib/--vram-kv-mib pair removed (PR #30); env, serve script and bats carried the new shape while the page and the AGENTS.md block still described the 1.0.13 3-slot era, so every point-of-use number was wrong. Re-derived from the live checkout and the 2026-10-19 11:10 boot ledger, not from the stale pre-#31 draft: runtime block (digest, version oracle, the two CLI breaks that are boot refusals), shape table (demand-signal rows, derived VRAM split, restored spec/batch row, host-only prefix tier, step sharing), container contract (no /data mount), the measured 2-slot ledger, a new Eras row for the retired 1.0.13 era whose rollback is a digest plus a trio rewrite, and the laws: pool sizing and the per-card budget re-scoped to the pair era, a new resolver law (funds ~90% of the worst case, host RAM below a slab step is inert, MAX_SEQS is a demand signal), the sludge law marked measured-at-3-slots with the fix ordered-not-verified, the knob-availability law flipped, and the slots law now pricing the third slot in KV (4.20 -> 2.76 GiB).
513 lines
45 KiB
Markdown
513 lines
45 KiB
Markdown
---
|
||
type: reference
|
||
title: "frank-archon on Dankserver — radiance (StillDeadcode) 1.8.1, publisher Qwen3.8-Flash-Next fp8-iq4r-moe .rad, 2× R9700 TP=2 (ACTIVE)"
|
||
description: "The radiance endpoint: live shape and harness contract, every era with its rollback path, and the laws it taught."
|
||
tags:
|
||
- llm
|
||
- frank-archon
|
||
- dankserver
|
||
- radiance
|
||
- gfx1201
|
||
- r9700
|
||
- tp2
|
||
- expert-offload
|
||
- endpoint
|
||
timestamp: 2026-10-19
|
||
---
|
||
|
||
# frank-archon — radiance (StillDeadcode), 2× R9700 TP=2, Dankserver :7770
|
||
|
||
The radiance engine (C++/HIP, `libr4d` kernels built for gfx1201, no Python in the serving
|
||
path) driving `podman run` straight from the digest-pinned image under a **root** systemd
|
||
unit; container name = unit name. Host Dankserver (192.168.0.235), port 7770, both Radeon AI
|
||
PRO R9700s. Capacity contract **2 conversations × 200,000 tokens at fp8 KV** — at 1.8.1 that pair
|
||
is the **demand signal** the budget resolver sizes the cache against, not a KV allocation (Laws).
|
||
A payload swap is a `MODEL_FILE` line plus a re-download — see Eras.
|
||
|
||
## Current shape
|
||
|
||
Authoritative source: `scripts/frank-archon.env` + `scripts/serve-frank-archon.sh` (the env
|
||
header carries the measured arithmetic beside each number).
|
||
|
||
**Payload** — `/home/dank/models/rad/qwen3.8-next-flash-fp8-iq4r-moe.rad`, 121,969,901,568 B
|
||
= **116,322 MiB**, sha256 `0af5e962…` (the Hub LFS oid), from
|
||
`StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe`: the engine author's own rad-convert of the
|
||
**official (censored)** `Qwen/Qwen3.8-Flash-Next` bf16 checkpoint. Encodings (recipe
|
||
`qwen4exp-w4nl64-i8-hc8m`): routed experts as 4-bit codes in a 16-level non-uniform codebook
|
||
(128-pt Walsh–Hadamard rotation, GPTQ on a 10M-token calibration + 3 coordinate-descent
|
||
sweeps, E4M3 scale per 64 weights); **10 whole bf16-protected experts per layer**; int8 W8A8
|
||
trunk + lm_head; fp8 HC mixtures; PLE n-gram fp8 at the checkpoint's fixed scale (47.7 GiB)
|
||
MTP head and the bf16 27-layer vision tower **in the container**. Card quality vs bf16: mean
|
||
KL 0.0789 all-tokens / 0.0239 assistant turns, top-1 92.85 %. The abliterated family lives on
|
||
[[frank-dragoon]]. Fetched by `scripts/download-frank-archon-iq4r.sh --fetch` (hfget resume +
|
||
declared size + sha256) as the transient unit `iq4r-fetch` (log
|
||
`/home/dank/logs/iq4r-fetch.log`, `…rc` sentinel); the staging dir
|
||
`/home/dank/models/StillDeadcode/…` keeps card, LICENSE and recipe as provenance, and only
|
||
`.rad` files are mounted.
|
||
|
||
**Runtime** — `stilldeadcode/radiance@sha256:d590e300234bf6d2f942a0b8d456d183153fd808f28820e6c2fa69c9389178ed`
|
||
= pin **1.8.1** (adopted 2026-10-18). The version oracle is the boot's own `STARTUP REPORT --
|
||
radiance 1.8.1` line, because `podman inspect` reports only the digest. The image lives in dank's
|
||
**rootless** storage and the host needs nothing but the amdgpu driver. Pin the INDEX manifest
|
||
digest, never the image ID (`69d05724…` is the config digest and sends `podman run` to the
|
||
network). The serve script's gate is that digest ref itself — `podman image exists name@sha256:…`
|
||
in one command (the 2026-09-30 no-pipe/grep-race lesson) — so a retagged look-alike or a newer
|
||
`latest` cannot satisfy it. 1.2.0 is a rejected serve pin (Laws). **Two CLI breaks ride this pin,
|
||
and both are boot refusals, never silently-ignored flags:** the `--vram-weights-mib`/`--vram-kv-mib`
|
||
pair died at 1.3.0 (the resolver derives the split) and `--prefix-cache-dir` /
|
||
`--prefix-cache-disk-mib` died with the disk tier. A rollback to 1.0.13 (`sha256:4c3f4845…`) is
|
||
therefore a trio rewrite, not a digest swap — the Eras row carries it, and
|
||
`tests/frank_archon_unit.bats` asserts the retired knobs are ABSENT from env and argv so a stale
|
||
line cannot reach a boot.
|
||
|
||
**Trio** — `scripts/frank-archon.env` (every number, with its arithmetic in the header)
|
||
`scripts/serve-frank-archon.sh`, `scripts/frank-archon.service`. Gates run in order: image
|
||
digest present → `.rad` present and ≥ `RAD_MIN_MIB` (a truncated convert would otherwise buy a
|
||
guaranteed long-timeout boot instead of this fast failure) → template present → `/dev/kfd` +
|
||
`/dev/dri` → `ttm_ensure_cap` (the serve script re-verifies the cap after the unit's hook) →
|
||
`podman rm -f` over `FRANK_ARCHON_LEGACY_CONTAINERS` (`frank-archon-qwen frank-archon-r9v
|
||
qwen-fp8 glm-probe`) plus its own name, because systemd SIGKILLs conmon on a crash and leaves
|
||
the name registered, and every one of those identities holds this GPU. Container contract
|
||
(publisher-vetted, `docs/DOCKER.md`): `--network host`, `--init` (so the engine releases the
|
||
cards on SIGTERM), `--security-opt seccomp=unconfined` (the default profile refuses io_uring —
|
||
the PLE table's reads — and the NUMA calls the ROCm runtime makes for host memory), `--ulimit
|
||
memlock=-1:-1`, `/models` ro + the rad-tune cache at
|
||
`/opt/radiance/share/radiance/tune` + the template as a single-file mount (there is **no `/data`**
|
||
any more: it left with the prefix disk tier, and `FRANK_ARCHON_STATE_DIR` is now only the host-side
|
||
directory that holds `tune/`), then `exec podman wait`: the script's lifetime IS the
|
||
container's, so a crash is a nonzero `ExecStart` and the restart policy is real.
|
||
Unit: `After=network-online.target`, `WantedBy=multi-user.target`, `User=dank`
|
||
`LimitMEMLOCK=infinity`, `StartLimitIntervalSec=0`, `OOMScoreAdjust=-500` (the last process the
|
||
OOM killer should pick), `Restart=always` **not** `on-failure` — a clean API-server exit after a
|
||
worker fault reads as success to systemd (the 2026-10-01 vLLM lesson) — and
|
||
`TimeoutStopSec=180` > the `ExecStop=podman stop -t 120` grace, because systemd's 90 s default
|
||
cut the stop short and recorded `Failed with result timeout` (measured 2026-10-01 13:27:17). No
|
||
drain contract is claimed: the OpenAI front end holds no queue, so an in-flight decode is lost
|
||
on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_period`).
|
||
|
||
| Knob (env name, `FRANK_ARCHON_` dropped) | Live value |
|
||
|---|---|
|
||
| TP / wire | `TP=2`, `TP_WIRE=wht6` — libr4d one-shot all-reduce, not RCCL; `min-kb` stays the engine default |
|
||
| Slots / window | `MAX_SEQS=2`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) — this pair is the **cache bound**, not a KV allocation |
|
||
| KV dtype | `KV_DTYPE=fp8` — the only dtype that funds the slot count (Laws) |
|
||
| VRAM split | **derived at boot** from `GPU_HEADROOM_MIB=96` + the cache bound + `HOST_POOL_MIB`; naming `VRAM_WEIGHTS_MIB`/`VRAM_KV_MIB` is a boot refusal (removed at 1.3.0) |
|
||
| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=44928` **per rank** |
|
||
| PLE | `NGRAM_PLACEMENT=ram` (47.7 GiB pinned; refuses boot on a short memlock) |
|
||
| Spec / batch | `SPEC_TOKENS=2` — MTP depth 2, the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); `MAX_BATCHED_TOKENS=2048` (MNBT — 2560 collapses decode, Laws) |
|
||
| Prefix cache | `PREFIX_CACHE_MIB=8192`, pinned **host tier only** — the 64 GiB disk tier is gone, so an evicted prefix is re-prefilled; `STATE_DIR=/home/dank/radiance-state` now exists only to hold the rad-tune cache |
|
||
| Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it |
|
||
| Step sharing | `BUSY_PREFILL_CHUNK=auto` (a shared chunk reserves a row plus the draft window per waiting decoder) · `PREFILL_INTERLEAVE=0` — held at zero until the decode-vs-prefill buckets ask for it |
|
||
| TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` |
|
||
| Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) |
|
||
|
||
Boot ledger for the live shape (2026-10-19 11:10, pin 1.8.1, `MAX_SEQS=2`, `HOST_POOL_MIB=44928`):
|
||
the resolver prints `elastic 26.73 GiB` → **experts 23.73 GiB (89 %) / kv cache 2.76 GiB (10 %)**,
|
||
`vram_weights 27.31 GiB`, `host_pool 44.47 GiB`. `kv_attn` (13 layers, 4-token blocks) pages
|
||
**26.00 KiB** → 91,301 blocks = **365,204 tokens = 1.8 full-length sessions = 91 % of the
|
||
2 × 200,000 worst case** (the resolver never funds 100 %, see Laws); `mover` lends 2,146 slots
|
||
(2.54 GiB) out of the cache to the expert slab; sampler **9 rows**, template 28,234 B, vision live
|
||
(≤ 4,194,304 px, 16,384 patches/pass).
|
||
|
||
The boot **ledger**, not this arithmetic, is the referee: radiance states every budget before
|
||
anything is allocated (weights vs KV, the host pool's coverage of the worst case, the PLE pin,
|
||
the P2P answer per pair) — read it after every resize, and unlike every predecessor there is no
|
||
per-boot VRAM planner to second-guess. For comparison, the publisher's own suggested run line on 32
|
||
GB cards was `--host-pool-mib 12288 --expert-vs-cache-ratio 0.82` with seqs 8 and spec 3 — a
|
||
1.0.x-era line, and `--expert-vs-cache-ratio` was removed at 1.2.0, so today it is a capacity floor
|
||
quoted from the card, not a run line this pin accepts; the
|
||
shape above is the fleet's measured one. `FRANK_ARCHON_EXTRA_ARGS` is the boot-ladder escape
|
||
hatch (unset → no extra argv; it is expanded unquoted on purpose, so keep multi-flag values as
|
||
one string).
|
||
|
||
**Harness contract** (`/home/blank/.omp/agent/models.yml`) — provider `llamacpp (dankserver)`
|
||
→ `http://192.168.0.235:7770/v1`, `api: openai-completions`, `auth: none`, `id: frank-archon`
|
||
(= `FRANK_ARCHON_SERVED_NAME`; the provider *label* is legacy and renaming it breaks
|
||
`config.yml` references). `contextWindow: 200000` must equal `MAX_MODEL_LEN`
|
||
`maxTokens: 16384` (clamp `min(maxTokens, window − estimate − 4096)`; compaction fires at
|
||
85 % → 170,000 est × 1.03 + 16,384 ≤ 200,000); `input: [text, image]`
|
||
`compat.reasoningContentField: reasoning_content`; `reasoningEffortMap` minimal/low→low,
|
||
medium→medium, high/max→xhigh; `extraBody` temperature 1.0 / top_p 0.95 / top_k 20 / min_p 0 /
|
||
presence_penalty 0 / repetition_penalty 1.0 + `chat_template_kwargs {enable_thinking,
|
||
preserve_thinking}`, every key inside the engine's known-parameter list (unknowns 400).
|
||
`compactionModel` routes to `llamacpp (dankclaw-3090)/frank-zealot`.
|
||
|
||
**Deploy / verify** — `./scripts/deploy.sh dankserver --service frank-archon`; launch it
|
||
detached when the swap restarts the endpoint your own session runs on. Then verify
|
||
**behaviourally**, not by the green gate: `podman inspect frank-archon --format '{{json
|
||
.Config.Cmd}}'` and `/v1/models` — `deploy.sh` plans from repo paths and gates on the port, so
|
||
host-side unit drift is invisible to it.
|
||
|
||
## Eras
|
||
|
||
One row per retired era on this port. The **Rollback** cell is the load-bearing one: anything
|
||
marked "re-fetch" is gone from `/home/dank/models/`, which today holds only `StillDeadcode`
|
||
`rad` and `tpl.jinja`.
|
||
|
||
| Window | Payload | Runtime | Shape | Replaced because | Rollback |
|
||
|---|---|---|---|---|---|
|
||
| 2026-10-15 → 10-19 | same `qwen3.8-next-flash-fp8-iq4r-moe.rad` (payload unchanged) | radiance **1.0.13**, `sha256:4c3f4845…` | 3 × 200,000 fp8 KV bought with the **explicit** pair `--vram-weights-mib 25600 --vram-kv-mib 5120`; host pool 44,928 → 45,568 MiB a rank; prefix cache 8 GiB host + 64 GiB disk | decode turned to sludge beside any long prompt — step contention, 18.7 ms decode-only steps against 317 ms steps carrying a prefill chunk (Laws) — and the only two knobs that fix it, `--busy-prefill-chunk` and `--prefill-interleave`, exist only after this pin | `FRANK_ARCHON_IMAGE` back to `sha256:4c3f4845…` **plus a trio rewrite**: restore the pair and the host+disk prefix knobs, delete `BUSY_PREFILL_CHUNK`/`PREFILL_INTERLEAVE`. A digest swap alone is a boot refusal (Runtime) |
|
||
| 2026-10-08 → 10-15 | in-house `qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB (Swift1.5 abliterated BF16, GPTQ w4 experts + 24 bf16 protected, KLD 0.0131 / top-1 97.92 %) | radiance 1.0.13, same digest | 3 × 200,000 fp8 KV, slab 25,600 / pool 5,120 — the shape the live file still carries | the engine author's own container is the supported path and his quality ceiling; the finetune's refusal-bending left the endpoint | `.rad` purged 2026-10-17 (trash-staged then emptied, Frank's approval). Re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` → `qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`, via the iq4r fetch script's hfget/verify pattern retargeted at that repo; point `MODEL_FILE` at it, redeploy. Re-**convert** is dead: the BF16 source was `rm`'d 2026-10-15 (`kld-ref`, grams, corpora and `/home/dank/logs/swift15_*.log` survive) |
|
||
| 2026-10-05 → 10-08 | `qwen4exp-fp8-native.rad`, 177,109 MiB (173.0 GiB) — native modelopt block-FP8, no requantisation | radiance 1.0.7 line, same digest | — | native FP8 experts (56.4 GiB/rank) ate a 40 GiB pool and left KV at 3,543 MiB = 386,904 tokens, **1.93 sessions** — context economics, not quality; and the host pool could not be trimmed to pay for it (36,864 refused the boot) | rebuild-only: `scripts/download-frank-archon.sh` (official `Qwen/Qwen3.8-Flash-Next-FP8`, 131 shards, 172.8 GB, ungated) + `scripts/build-frank-archon-rad.sh` (no-recipe convert under the descriptor law); raw **and** `.rad` were `rm`'d 2026-10-15 (Frank's call, to free the volume for the iq4r download) |
|
||
| 2026-10-05 (same day) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` safetensors, 131 shards | FreeToken PR #587 head `77a7201` (`rocm714_torch214`), image `localhost/frank-primary-freetoken:pr587-77a7201` | — | TP=2 cannot boot on ROCm: the per-rank `pynccl.cu` JIT cannot resolve `cuda_runtime.h` for gfx1201, so the second card idled | git tag `frank-archon-freetoken-pr587-20261005` (trio + `scripts/docker/frank-primary-freetoken.Dockerfile`, which exists only there); image still in rootless storage; payload needs a re-fetch, and that repo 403s until the gate is accepted on the account |
|
||
| 2026-09-29 → 10-05 (interrupted 09-30 → 10-01 by GLM) | same orcarouter FP8 safetensors | tcclaviger vLLM fork, `docker.io/tcclaviger/vllm@sha256:da402966…` | shape G: 3 × 200,704 fp8, KV 3,166,500,000 B/rank → 401,408 tokens, arena 17.77 GiB / 7,762 slots, util 0.982, MNBT 2048, no spec, no prefetch, prefix caching ON | radiance gave real TP=2 (no RCCL) and a 2-3 min boot, and the fork never fixed its long-context class | git tag `frank-archon-vllm-shapeG-20261003` (trio + the 362-line env lore); image still in dank's rootless storage; payload needs a re-fetch. The 2026-10-01 fp8+MTP4 rung is tag `frank-archon-fp8-mtp-20261001` |
|
||
| 2026-09-30 → 10-01 | `Intel/GLM-5.3-Flash-W4A16-AutoRound` ~170 GiB (sym int4 g128, `auto_gptq`) | same tcclaviger digest; `R4dW4a16MoEMethod` → libr4d grouped W4A16 | — | — | payload (170 GiB) + drafter (2.2 GiB) went to `~/.AGENTTRASH/20261001-glm-purge/` and were emptied; the cutover was reversed by `d91379c`, restoring the `2339fae` trio; `scripts/patches/glm53-*.patch` survive in-tree but none applies byte-exact since the 2026-10-10 rename rewrote their context lines |
|
||
| 2026-09-27 (same day) | `Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS`, 92.39 GiB + PLE 26.82 GiB (`34fa36f8…`) | same R9V bundle + shim | `MAX_MODEL_LEN=196608`, offload budgets 146.25 / 78.75 GiB, PLE residency `ssd` mmap, tool parser `qwen3_coder`, thinking on the `reasoning` (vLLM) channel | same-day flip back to the reference quant with the MTP4 head, at Frank's direction | `/home/dank/r9v-state/uncensored/` (`setup.json`, `restore-config-20260927.env`) is still on disk; the IQ4_XS payload was deleted in the 2026-10-17 cleanup → re-fetch, point `R9V_STATE_FILE` back, drop the two q8-only keys and the template keys |
|
||
| ≤ 2026-09 — 4 earlier eras: payloads, shapes and every measured number destroyed 2026-10-19 as obsolete | — | — | — | — | —
|
||
|
||
Before 2026-09-26 the port belonged to the **`frank-medium`** service, and its era
|
||
pages are **not** superseded here: is the locator for that
|
||
endpoint's whole shape history, including the unresolved end date of the
|
||
vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
|
||
|
||
## Laws learned here
|
||
|
||
### The radiance engine
|
||
|
||
- **KV dtype law.** `--kv-cache-dtype` takes only `bf16|fp8` — ONE dtype for the whole cache,
|
||
a **radiance** restriction, not stock vLLM's: upstream advertises `auto|fp8|fp8_e5m2|fp8_e4m3`
|
||
(`docs.vllm.ai` engine-args) and even there the advertised set is not the usable set — several
|
||
accepted values crash engine init at the attention layer's own dtype check
|
||
([vllm-project/vllm#42587](https://github.com/vllm-project/vllm/issues/42587)).
|
||
So llama.cpp's exact-K / quantized-V pair (what [[frank-zealot]] runs as `--cache-type-k`/
|
||
`--cache-type-v`) is unservable here: the rung below bf16 is fp8 on both halves. Measured
|
||
per-rank cost out of the boot ledger's group table (fp8 boot 2026-10-08, bf16 boot
|
||
2026-10-11): `kv_attn` 13 layers / 4-token blocks pages **26.00 → 52.00 KiB**, while
|
||
`kv_qsa_bkey` stays **3.25 KiB at both** — the QSA block key is an index, not attention state,
|
||
and does not follow the flag. So 7.3125 KiB/token at fp8 vs 13.8125 at bf16: **1.89×, not
|
||
2×**, precisely because the bkey plane is dtype-fixed. Secondary saving: a prefix-cache entry
|
||
tracks the attention plane (110.50 KiB at bf16 → 58.50 at fp8, per the `kv tier transfer`
|
||
lines), so the same 8 GiB host tier parks ~5.8 sessions of 200k at fp8 where it parked ~2.9 at bf16.
|
||
- **Pool sizing law (the explicit-pair era — at 1.8.1 the cache is derived, so the resolver law below
|
||
owns it).** `pool_mib ≥ tokens × 7.3125 KiB + 110 × seqs + 448` — the dtype-
|
||
independent terms are also in the pool: `kv_gdn_state` 54 MiB × 2 copies per sequence (+conv/
|
||
tail/ple ≈ 110 MiB a sequence) and the prefix checkpoint region 8 × 55.99 MiB. Deployed: 3 ×
|
||
200,000 at fp8 = 4,285 + 330 + 448 = **5,063 MiB**, pool 5,120, which predicts (5120 − 330 −
|
||
448) × 1024 / 7.3125 = **608,027 tokens** against a 600,000 contract (1.3 % slack) — every
|
||
MiB above 5,063 was handed to the expert slab instead. At bf16 the same three slots cost
|
||
8,093 + 778 = 8,871 and the slab falls to 20,249 (−9 %); four slots at bf16 cost 10,791 + 888
|
||
= 11,679 and the slab falls to 17,441 (−33 %); fp8 keeps 22,208 of slab at both slot counts.
|
||
**fp8 is not a preference, it is what funds the slot count.** Measured backings to sanity-
|
||
check the formula: bf16 pool 6,912 at seqs 2 backed 462,752 tokens (2026-10-11), the fp8-era
|
||
pool of 3,543 gave 386,904 (1.93 sessions), and the four-slot fp8 boot of 2026-10-13 19:21
|
||
backed 843,276. The ledger's `backs N sequences all at 200000 tokens at once` line is the
|
||
referee — a pair this tight is read off the boot, never assumed, and the concurrent probe is
|
||
its regression test. A fourth full-height conversation needs 6,601 MiB of pool: refusing it is
|
||
Frank's explicit call, and under that pin serving it was a `--vram-kv-mib` change, not a code change.
|
||
At 1.8.1 the same decision is `MAX_SEQS` — the resolver buys the cache the demand signal implies.
|
||
- **The hard limit is per request.** The engine treats `kv group tokens >= max-model-len` as a
|
||
HARD limit — "one request longer than N tokens is accepted by the API and then refused as too
|
||
large to ever fit — this is a hard limit, not a backing ratio" — so the pool is sized for
|
||
`seqs × max-model-len`, not for one sequence. Radiance never truncates, at any size.
|
||
- **One VRAM budget per card** (the arithmetic below is the pair era's; the ceiling survives, the knobs
|
||
do not): `weights + kv ≤ 30,969 MiB`, read off the engine's own budget
|
||
table (boot 2026-10-13 at pair 24000/5120): card 32,624 MiB, budgets claimed 29,120, and
|
||
"*3.42 GiB is neither budgeted nor accounted — the driver's context, the activation arena and
|
||
`--gpu-headroom-mib` come out of it; anything past those is throughput on the floor*", with
|
||
the card settling 2.01 GiB free against a 209.19 MiB floor (96 headroom + 113.19 kept for the
|
||
static region). Reserve above the engine's floor = 2,058 − 209 = 1,849, ceiling 29,120 + 1,849
|
||
= 30,969; the deployed pair lands 458 MiB above that floor. The 2026-10-11 bound of 29,120
|
||
("31,856 − 1,710 outside both budgets − keep 1 GiB free") was a hand-picked reserve, not the
|
||
engine's wall. Under that pin, boot with the explicit `--vram-weights-mib`/`--vram-kv-mib` pair,
|
||
never `--kv-cache-size-mib`; with the disk tier engaged the engine also forced ~9.6 GiB of weight
|
||
buffers + arena onto the weights side (~29.6 GiB/card effective). **Neither flag exists at 1.8.1:**
|
||
the same ceiling is enforced by the resolver, and passing the pair is a boot refusal.
|
||
- **At 1.8.1 the split belongs to the resolver, and it funds ~90 % of the worst case, never 100 %.**
|
||
The engine takes `--gpu-headroom-mib`, the cache bound (`--max-model-len` × `--max-num-seqs`) and
|
||
`--host-pool-mib`, prints `claimable → elastic → experts / kv cache` before allocating anything, and
|
||
sizes the expert plane from the model's own weight footprint plus slack — not from a number typed by
|
||
hand. Three boots of evidence:
|
||
|
||
| shape | demand | cache given | backing |
|
||
|---|---|---|---|
|
||
| 3 × 200,000, host pool 44,928 | 600,000 tok | 4.20 GiB | 93 % (2.8 sessions) |
|
||
| 3 × 200,000, host pool 45,568 | 600,000 tok | **4.20 GiB — byte-identical** | 93 % |
|
||
| 2 × 200,000, host pool 44,928 (live) | 400,000 tok | 2.76 GiB = 365,204 tok | 91 % (1.8 sessions) |
|
||
|
||
So `--max-num-seqs` is a **demand signal** for the resolver and an admission cap for the scheduler —
|
||
never a KV allocation — and no slot count fully funds the worst case. Growing host RAM by less than a
|
||
slab step (640 MiB) moved nothing: the expert plane's claim decides what is left for KV, so cache is
|
||
grown through slots × window or the dtype, and experts are grown by not leaving elastic pool idle.
|
||
`--host-pool-mib` is still load-bearing the other way — it is where the expert rows the cards cannot
|
||
hold get pinned, and under-sizing it has always been a boot failure (36,864 refused a boot in the fp8
|
||
era). The two-slot boot's own behavioural check: three concurrent conversations all answer `200` —
|
||
the third **queues** (`queue_cap: 16`), it is not refused; the 400 class is a single request over
|
||
`--max-model-len`, or three concurrent full-height conversations against a 1.8-session cache.
|
||
- **Idle KV VRAM is the expert cache's spillover — the price is paid twice.** `mover: 2275
|
||
slots can be lent out of the KV cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <=
|
||
256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 in the pair era)
|
||
so a bigger KV pool steals weight residency from the slab AND from the lendable cache. Priors
|
||
on this card/model: stealing 2.14 GiB of weights to buy `MAX_MODEL_LEN 262144` cost 2.5 %
|
||
decode; the other direction (`--expert-vs-cache-ratio` 0.82 → 0.84, +2.5 GiB of weights) cost
|
||
8.9 %.
|
||
- **A KV-pool cut is a runtime-headroom lever, never an arena lever.** The 6.1e9 → 3.15e9 B/rank pool cut
|
||
(2.75 GiB/rank) bought **zero** extra expert-cache slots. The arena moves with the reserve, one-for-one
|
||
(d(reserve) 0.57 GiB = d(arena) 0.57 GiB = **+248 slots**). The shape that pool cut replaced died with
|
||
`0 bytes free` on a real 995-token request — that is headroom, not slot count. Read `expert cache space … / N
|
||
slots` out of the boot ledger before trusting any pool arithmetic: the 2026-10-13 read was **14.87 GiB /
|
||
6,496 slots** against `GPU KV cache size: 398,658 tokens`, and it is the POOL that caps resident
|
||
conversations, not the arena.
|
||
- **MNBT moves the compile RANGE, not the batch width.** The engine sets `num_reqs = min(max_num_batched_tokens,
|
||
max_num_reqs)`, so the CUDA-graph compile width is **SEQS + 1** for any `MNBT >= SEQS` — 3 at SEQS 2, 4 at
|
||
SEQS 3, exactly the constant printed in the engine's own message. MNBT only shifts the compiled range
|
||
((1,1024) versus (1,2048)), which is why the identical capture failure appears at both rungs;
|
||
`--enforce-eager` bypasses the path and loses CUDA graphs with it.
|
||
- **The prefix cache proves itself at depth, not in the ledger.** Two concurrent conversations of 196,919 tokens
|
||
each answered the marker buried at the head of **their own** prompt with **393,837 tokens resident** — above the
|
||
fp8 pool's 386,904, so the answer came from a cached prefix and not a re-prefill. Same boot (2026-10-13):
|
||
decode at 51.5k depth **94.1 cold / 102.3 warm / 117.6 prefix-hit t/s**, draft acceptance **0.7284 / 0.7525**.
|
||
- **The host pool moves with the slab.** `vram_weights` and `host_pool_mib` serve the per-rank
|
||
resident set *together*: at the fp8-KV pair (26071/3543) the ledger showed the slab full to
|
||
259 KiB of slack, and cutting `vram-weights` by 3.8 GiB to pay for a bf16 pool spills ~3.8
|
||
GiB/rank into host RAM — under-sizing the pool there is a **boot failure** (36,864 refused the
|
||
boot in the fp8 era). The ledger's coverage line, not the arithmetic, arbitrates.
|
||
- **Slab → prefill: the arms disagree, the cliff does not.** 2026-10-11 exact-KV A/B (slab
|
||
22,208 vs the fp8-era 26,071, −14.8 %): prefill −29 %, TTFT at the full window +25 %, decode
|
||
inside the noise band on 2 of 3 passes, draft acceptance UP 0.661 → 0.740. 2026-10-13
|
||
(`probe-frank-archon-kv.py`, env header): A-cold 43.5k 22,208 → 3,397, 24,000 → 3,290, 25,600
|
||
→ **3,584** t/s, and the 3-way 197k arm's aggregate 3,051 → 3,589 (wall 193.7 → 164.5 s), so
|
||
+1,600 MiB of slab bought +8.9 % single-stream / +17.6 % concurrent — read then as a SLOPE,
|
||
the 22,208/24,000 pair having landed inside noise of each other and falsely looked like a
|
||
plateau. Re-measured at 3 slots after the 2026-10-15 swap the 25,600 arm returned 3,290: the
|
||
3,584 did **not** reproduce, so treat 22,208-25,600 as one band until someone re-runs the
|
||
ladder. What is not disputed is the cliff: 26,071 → 4,379 t/s (the 2026-10-11 control at seqs
|
||
3 / pool 3,543), +22 % for +471 MiB — unreachable at 3 slots (26,112 + 5,120 = 31,232 vs
|
||
30,969) and free at 2 (pool 3,072 + slab 27,897 = the ceiling exactly). Two points inside
|
||
noise of each other are not a plateau, and two points are not a curve.
|
||
- **Decode is nearly window-free; slots are paid in concurrency.** 99.0 t/s at 43.5k vs 93.3 at
|
||
197k (−6 % for 4.5× depth), while three streams give 113.9 / 43.3 / 10.9 = 18.4 t/s aggregate
|
||
— a fused-step lottery, not a fair share. Read it from leg D of `probe-frank-archon-kv.py`
|
||
with ≥ 128 generated tokens; an 8-token probe measures first-token noise.
|
||
- **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams
|
||
in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3
|
||
GiB, refused).
|
||
- **Decode pays for the chunk it shares a step with.** Census of the live engine taken at the
|
||
3-slot shape on 1.0.13 (`/stats` + `/metrics`, cumulative over the 2026-10-15 boot):
|
||
1,435,585 decode-only steps at **18.7 ms** each against 127,991 prefill+mixed steps
|
||
at **317 ms** — 60 % of engine wall-clock spent inside 8 % of the steps, because the
|
||
chunk is `min(remaining, checkpoint interval 2048, MNBT 2048)` = 2048 rows and every
|
||
decode row in that step waits the whole chunk. Day of Prometheus (120 s step, 721
|
||
samples): `decode_tokens_per_second` median **13.3 t/s** in the 56 % of samples where
|
||
`prefill_tokens_per_second` ≥ 1000, against 106-240 t/s in the light-prefill buckets;
|
||
bucketed by mean step time it is 128.7 t/s at 0.02-0.05 s steps and 12.0 t/s at
|
||
0.15-0.4 s. The stall is **during**, not after — decode in windows 2-14 min after a
|
||
busy-prefill window sits at the idle baseline (58-64 vs 77-85 t/s, the residual
|
||
confounded by co-scheduling), so no lingering expert-residency penalty at 2-minute
|
||
resolution, and `preemptions: 0` rules out preempt-and-re-prefill. Method:
|
||
[[LLM Performance Diagnosis]] §9. The law is shape-independent; the fix lives at 1.8.1 (next law),
|
||
and nobody has re-run the census since the migration — so the fix is ordered, **not verified**.
|
||
- **The knobs for it arrived with 1.8.1, and only one of them is on.** `--busy-prefill-chunk` caps the
|
||
shared chunk so a step with waiting decoders takes a smaller piece (live: `auto`, the engine picks;
|
||
the explicit value is 512), and `--prefill-interleave N` inserts N decode-only steps after a chunk
|
||
step taken beside decoders. `interleave` is deliberately **0** — the env comment holds it there until
|
||
the decode-vs-prefill buckets ask for it, and those buckets have not been read since the migration.
|
||
The vendor prices a 1024 chunk at about a fifth off prefill throughput to halve the stall; that is a
|
||
knob for choosing tomorrow's chunk, not a measurement of this shape. MNBT remains an arm and not an
|
||
edit: 2560 collapses decode through MTP resonance. Any chunk change re-scores draft acceptance before
|
||
it is trusted.
|
||
- **Slots cost KV, rows, and window — never speed.** `--max-num-seqs` carries no radiance constraint: it
|
||
adds ≈110 MiB of recurrent state per slot (the live `kv_full_backing` arithmetic prices them exactly),
|
||
declares the fused TP all-reduce over `seqs × (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows
|
||
(single-sequence decode and any step of ≤ 25 rows is exact; only busier steps are lossy) — 2 × 3 = 6
|
||
today. Measured 2026-10-06: seqs 2 and seqs 3 are statistically tied (30.9-33.1 t/s, inside the ±3 %
|
||
floor). Measured 2026-10-19 at 1.8.1: the third slot's real price is **KV, not rows** — dropping it
|
||
moved the resolver's cache from 4.20 GiB to 2.76 GiB and the backing from 2.8 to 1.8 sessions, while
|
||
the ≈4 GiB of host RAM it freed came back unbudgeted, landing in elastic pool that turned into a bigger
|
||
mover lend instead of cache (resolver law above). The old "seqs 2 cannot boot" law belonged to the
|
||
retired tcclaviger fork's `torch.compile` shape specialization, and radiance booted seqs 2 on its first
|
||
fp8-native start.
|
||
- **A dequantize-to-compute path bills host RAM twice.** On the EMULATION backend an offload request
|
||
of **118 GiB** landed as **~238 GiB** of real host RAM — the dequantized copy rides on top of the
|
||
pinned weights, and the doubling is invisible in the weight count. Read the engine log for
|
||
`Using 'EMULATION' Mxfp4 MoE backend` before concluding "doesn't fit"; a page that says a payload
|
||
"does not fit" without that line has not read the runtime.
|
||
- **One encoding per expert table** (`rad_arch_declare`): a whole expert kept bf16 is the only
|
||
thing legal beside the table, so a split tier (down=fp8, gate_up=w4) is unservable.
|
||
- **Descriptor-selection law.** The plugin is picked by the exact `(arch_id, quant descriptor)`
|
||
pair with no fallback: a raw official FP8 checkpoint (`quant_method: "fp8"`) is refused by
|
||
serve *and* convert, because the shipped `qwen4exp_fp8` plugin claims `(qwen4exp, "")`. Stage
|
||
a `config.json` with `quantization_config` dropped — the descriptor changes, the data stays FP8.
|
||
- **Three convert laws, each paid in a full verify+declare cycle.** Scale-name: `scale_name()`
|
||
appends `_scale_inv` to the *tensor* name, so stacked `…experts.gate_up_proj` keeps its scales
|
||
at `…gate_up_proj_scale_inv`. File-shadow: the reader opens index-named files sorted and
|
||
emplaces every tensor each file physically holds, first-wins — a merged PLE shard must sort
|
||
first (`00-ple.merged.safetensors`) or a raw shard wins the name. Symlink-through: staging
|
||
symlinks the raw index, so `open(path,"w")` writes through it and corrupts the checkpoint —
|
||
write temp + `os.replace`.
|
||
- **Convert is card-less** (`rad-convert` declares on CPU, GPTQ solves on CPU BLAS) and
|
||
`--plan-only` goes first: a plan that resolves every op will boot. `rad-tune` must be
|
||
declared at the shapes you serve and its cache lives in the *image* (`$RADIANCE_HOME/tune`)
|
||
so it only survives on the `$FRANK_ARCHON_STATE_DIR/tune` bind-mount the serve script adds.
|
||
- **Quant gates are scored at the production shape** (MNBT 2048, not the recording's 32768)
|
||
against the BF16 container: KL ≤ 0.03, top-1 ≥ 95 %, and the verdict is chunk-size invariant
|
||
(arm Δ ≈ the 0.0015 float-order band) and placement-invariant. Protected experts are
|
||
data-driven — energy share ≥ 3 % AND ≥ 3× routing share, even count per rank half — and the
|
||
PLE rides the checkpoint's own fixed scale (1.9931793212890625e-04, absmax inside the
|
||
2⁻¹³..2⁻¹⁰ plateau).
|
||
|
||
### This host, and the deploy path
|
||
|
||
- **The pinned-RAM ledger is why this is a root unit:** 2 × 44 GiB expert pools + the 47.7 GiB
|
||
PLE n-gram table (51B params, read from pinned host RAM over io_uring — Frank's pin law:
|
||
"*Native FP8. Experts from ram. Pin the PLE to ram.*") + the 8 GiB prefix host tier ≈ **147
|
||
GiB** charged against `RLIMIT_MEMLOCK`; dank's login session caps at 8 MiB and a user quadlet
|
||
is capped by the user manager, so only PID1 can grant `infinity` — and
|
||
`--ngram-placement ram` refuses the start when the limit is short.
|
||
- **TTM is runtime state, and the gate that owns it raises before it refuses.** The pinned pages
|
||
ride `/sys/module/ttm/parameters/pages_limit` (+ `pages_max` on kernels that expose one; this
|
||
kernel exposes **none** — the cap is the MIN over the knobs present, and an absent knob is
|
||
skipped, never read as zero, because a zero would refuse a healthy cap). The kernel default is
|
||
**half of RAM**, so every raise dies with the reboot: the 2026-10-05 manual raise to
|
||
52,428,800 pages (200 GiB) was gone after the **2026-10-12 23:17 power failure**, the cap fell
|
||
to 125,988 MiB, and the preflight refused the start **6,919 times over ~20 h** while
|
||
`Restart=always` cheerfully re-ran a refusal. A manual sysfs write is therefore not an
|
||
acceptable repair procedure — `scripts/ttm-preflight.sh` writes `TTM_TARGET_MIB` (204,800 MiB
|
||
= 52,428,800 pages; pages are 4 KiB so pages = MiB × 256; the *recorded host ceiling*, because
|
||
writing the 150,000 floor would leave the ledger no slack) into every knob present, re-reads,
|
||
and only then refuses. Write failures are swallowed deliberately: the caller re-reads, and ttm
|
||
itself refuses a limit below the pages it has already pinned.
|
||
- **Privilege of that knob, measured the hard way:** it is root-owned 0644 and the service runs
|
||
`User=dank`, where `-w` is false and a plain `echo` fails. The canonical privileged path is the
|
||
unit's `ExecStartPre=+…` (the `+` runs it as root even under `User=`); `sudo -n` is the
|
||
fallback for a human. The file is *sourced* by the serve script (which re-verifies the cap
|
||
before `podman run`) and *executed* by the unit. A refusal is invisible outside the journal —
|
||
no container is created, the unit sits in `activating (auto-restart)` on one repeating line —
|
||
and without the gate the load instead dies mid-pin with a bare allocator error.
|
||
- **The prefix host tier needs real RAM:** 8 GiB of it, and with the service loaded the host
|
||
still has ~104 GiB available (246 GiB total) — that headroom is what the pinned ledger is
|
||
measured against.
|
||
- **The deployer's blind spot is host-side unit drift.** A leftover
|
||
`frank-archon.service.d/bench.conf` overrode both `FRANK_ARCHON_ENV_FILE` and `ExecStart` to a
|
||
staging worktree and every `deploy.sh` since was green against the old configuration —
|
||
drop-ins are read *after* the unit's own `Environment=` lines, so they win.
|
||
- **Never `systemctl disable` a unit on these hosts** — it removes the `/etc/systemd/system/`
|
||
entry, which here is the symlink into the checkout. `mask` is reversible but *fails on a
|
||
relinked unit*, so a bench window uses `stop` and the deploy restores the unit.
|
||
- **The restore oracle for an experiment window is a conjunction:** `is-active` == active AND
|
||
`is-enabled` == enabled AND no `/run/systemd/system/<unit>` mask symlink AND the container is
|
||
owned by the unit and not `Stopping`. The vLLM/GLM-era engines ignored SIGTERM and answered
|
||
`/health` 200 for 20+ minutes while `Stopping`; a runtime mask survives `stop` and blocks
|
||
`start`, and a failed unit needs `reset-failed` first.
|
||
- **`MemAvailable` is an input, not a backdrop:** the expert-offload planner reads it, so
|
||
anything booted beside the live service dies at plan time (`ExpertPlanError: host holds 5113
|
||
of 12096 swappable experts`) — and fleet law forbids GPU experiments beside production
|
||
anyway. Long remote work runs as a transient unit or `setsid`+disown: `nohup X &` inside `ssh`
|
||
races ssh teardown and can die before writing a byte.
|
||
- **The ~40k-token corruption class was the RAM, not the engine.** Dropping DDR5 from 4800 to
|
||
4600 MT/s ended it, with zero `gdn_hip` faults either side; the prefix-caching / GDN re-seed
|
||
theory was confounded on both arms. Consumer non-ECC raises no MCE for a silent flip and a
|
||
60 s / 8 GiB pattern check covers 0.5 % of 246 GiB — silent RAM corruption outranks every
|
||
software theory.
|
||
- **TP=2 here** is legal since the 2026-09-20 ProArt B850 swap (x8/x8 on CPU root ports,
|
||
`Routing+`/`ReqEn+` end to end) only because radiance's wire is `libr4d`'s own one-shot
|
||
all-reduce; RCCL needs atomics on the whole chain and fails `hipErrorIllegalState` without
|
||
them (). `wht6` buys +5.6-6.3 % prefill for ~1.3 % of 8-stream
|
||
decode, `exact` is the bit-exact alternative.
|
||
- `--prefix-cache-dir` creation is **not recursive** — the serve script `mkdir -p`s the path or
|
||
the disk tier silently never opens. The tier does survive a payload swap when arch and
|
||
tokenizer match (it recovered 900,176 blocks on the 2026-10-15 swap; `cache_n` proves a hit).
|
||
|
||
### Dead ends — do not re-try
|
||
|
||
- **radiance 1.2.0 as the serve pin** (`sha256:665775ed1d90de6dbb25558e07fc60d20328f1346f96f3c6e00845eb763f01fa`
|
||
commit `e95d397`), benched 2026-10-06 on this exact shape with a 43.6k greedy probe: decode
|
||
29.21 t/s and draft acceptance 0.7123 vs the 1.0.13 control's 30.08 / 0.7549, and the control
|
||
reproduced the recorded 0.755 exactly — greedy acceptance is deterministic, so the drop is a
|
||
real numerics change (the decode loss tracks the acceptance ratio, 0.943 vs 0.945) with the
|
||
1.2.0 MoE weight-rotation change `92b8aaf` the named suspect. The vram-pair translation itself
|
||
is exact (the 1.2.0 ledger took 25.46 GiB weights / 3.46 GiB KV per card, byte-identical to
|
||
1.0.13 at EVC 0.82), and 1.2.0 REMOVED `--expert-vs-cache-ratio` in favour of the pair, which
|
||
both pins accept. Re-test 1.2.0+ once the rotation change is untangled.
|
||
- **FreeToken TP=2 on ROCm:** the `pynccl.cu` JIT cannot resolve `cuda_runtime.h` for gfx1201 —
|
||
an untested edge of that PR, not a board fault.
|
||
- **tcclaviger forks:** `--max-num-seqs 2` (`ConstraintViolationError` on `query_start_loc`)
|
||
`--gpu-memory-utilization 1.00` unbootable, 0.982 is the law; expert prefetch is poison (36 %
|
||
precision — deleting it was +40-51 % decode); TunableOp-with-offload corrupts numerics
|
||
(repetition loops, early EOS) — autotuning that changes text is broken tuning; ngram-only
|
||
drafting dies at init (`PLE inputs were not prepared`); DCP needs TP > `num_kv_heads` (2 here,
|
||
so 4 GPUs); 6 × 262,144 fits no KV dtype (6 sequences leave 9.18 GiB of pool where fp8 needs
|
||
12.47). fp8 KV on that path also removes the only MLA decode gfx12 has (the aiter op is gated
|
||
on CDNA > 2, so the draft step demands `…mla_decode_fwd`, which cannot exist there) and its
|
||
KV reserve model is dtype-blind (identical 13,600 B/token plan line) — no capacity for the breakage.
|
||
- **DeepSeek-V4.1-Flash on 32 GiB cards:** only `--ds4-prefill exact` is legal,
|
||
`--ds4-fused-decode` is refused, and at ~22 % resident experts each token pulls ~187 × 12 MiB
|
||
from NVMe — low GPU util with a pegged link is a *capacity verdict*. Two corrections survive:
|
||
expert TP **does** work on gfx1201+gfx1201, and an expert-placement JSON outside the mounted
|
||
`/models` tree gives a silent single-tier fallback.
|
||
- **Which experts you pin does not matter; how many does.** An IBS census of the live server
|
||
(`scripts/frank-archon-expert-activity.py`) measured per-block read volume flat at 0.93-1.08×
|
||
of mean across every CPU-resident block — top-k routing fires all of them every token, so at
|
||
`-ot` granularity the lever is COUNT.
|
||
- **Never mount a diagnostic hook on `PYTHONPATH` without a marker guard:** `sitecustomize` is
|
||
imported by every interpreter, a child inherits the hook and spawns again — a 19-level python
|
||
fork bomb inside the container. Kill with `podman rm -f` (the cgroup owns every generation)
|
||
never pattern-kill an interpreter name on this host; use `TORCH_LOGS` for dynamo questions.
|
||
|
||
## Contract (the wire)
|
||
|
||
- **Live surface:** `/health` (== `/ping`), `/v1/models`, `/v1/chat/completions`
|
||
`/v1/completions`, `/metrics`, `/stats`, `/sessions`, `/dashboard`. `/health` answers 200
|
||
JSON (model id, `in_flight`, `queue_depth`) while serving and **503 only while stopping**
|
||
(`core/server/admin.cpp:385`), and the listener binds *after* the full load (`main.cpp`:
|
||
`engine.start()` then `engine.serve()`) — during the load the port refuses connections, so a
|
||
bare `curl -fsS …/health` gate is honest and needs no body grep.
|
||
- **Served id.** `FRANK_ARCHON_SERVED_NAME=frank-archon` (the wire rename of 2026-10-09 moved
|
||
only the alias; container, unit, scripts and env keep the service name). The env's contract
|
||
line is that **radiance validates the requested model name, so every client — harness, bench
|
||
scripts, probes — MUST send it** — while the era pages also record that a chat response
|
||
*echoes* the requested name, which is why `/v1/models` is the served-id oracle and a 200 chat
|
||
probe proves nothing about what is actually loaded.
|
||
- **Reasoning** lands in `reasoning_content` (`--reasoning-format` defaults to `auto`; no
|
||
parser flags exist or are needed) and `usage` bills it separately (`reasoning_tokens`).
|
||
**Thinking consumes `max_tokens`**, so the classic false negative is a probe with empty
|
||
`content` and `finish_reason: length`/`stop` because the whole budget went inside the think
|
||
block: either set `chat_template_kwargs.enable_thinking=false` or budget a few hundred tokens
|
||
and read `reasoning_content`. Same trap, sharper, on the vision probe below.
|
||
- **Sampler surface:** `temperature, top_p, top_k, min_p, typical_p, presence_penalty,
|
||
frequency_penalty, repetition_penalty, repeat_last_n, chat_template_kwargs, reasoning_effort,
|
||
enable_thinking, preserve_thinking` accepted; **unknown parameters are refused 400** (nothing
|
||
unverified can ride the wire, and an `extraBody` typo fails loudly); `max_tokens` is clamped
|
||
not refused; `response_format: json_object` honoured; **`logprobs` refused**; no `seed` = the
|
||
engine picks; `n` defaults to `max(8, max-num-seqs)`; stop strings ≤ 64 × 4096 B.
|
||
- **Tools:** the qwen3.8 dialect from the chat template → `message.tool_calls` +
|
||
`finish_reason: tool_calls`. **`reasoning_effort`:** the froggeric template does not gate the
|
||
set (xhigh/high/medium/low all 200); the harness map is a safety net, not a requirement.
|
||
- **Vision is native** (the checkpoint's 27-layer Qwen-VL tower through the engine's Qwen-VL
|
||
processor): an `image_url` content part, no legacy top-level `images` field, ≤ 4,194,304 px
|
||
and 16,384 patches per pass. Probe with a ≥ 256 px solid-colour PNG and a ≥ 256 token budget
|
||
— at 64 tokens you get empty `content` and the colour only inside `reasoning_content`.
|
||
- **Capacity behaviour:** a prompt over `--max-model-len` is refused, never truncated; a request
|
||
longer than the KV pool's token count is accepted then refused as too large to fit; under the
|
||
3-slot contract a **fourth** full-height conversation is refused, not paged or squeezed.
|
||
- **Observability:** `/metrics` carries vLLM-compatible names plus `radiance:*`, scraped by the
|
||
Dankclaw Prometheus job at `192.168.0.235:7770` over plain LAN routing (the netavark
|
||
published-port quirk is same-host only). The boot ledger — the referee for every shape number
|
||
above — is in the container log (`podman logs frank-archon`)
|
||
`/home/dank/logs/frank_archon.log` is where the serve script redirects `podman run`.
|
||
|
||
## Files
|
||
|
||
- Shape and law owner: `scripts/frank-archon.env` (measured arithmetic in its header)
|
||
`scripts/serve-frank-archon.sh` (gates image / `.rad` size / template / GPU nodes, retires
|
||
stale containers, `podman wait`), `scripts/frank-archon.service`, `scripts/ttm-preflight.sh`.
|
||
- Payload doors: `scripts/download-frank-archon-iq4r.sh` (live), `scripts/download-frank-archon.sh`
|
||
+ `scripts/build-frank-archon-rad.sh` + `scripts/stack-qwen38-official-fp8.py` (the fp8-native
|
||
rebuild), `scripts/download-frank-archon-swift15.sh` (the BF16 source a w4dn8 re-convert
|
||
needs), `scripts/hfget.sh` / `scripts/hf-fetch.sh`; template
|
||
`scripts/qwen38-froggeric-v22.5.jinja`.
|
||
- Convert campaign (dormant, kept for a future quant): `scripts/convert-swift15-w4dn8.sh`
|
||
`scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`, `scripts/run-swift15-bf16.sh`
|
||
`scripts/kld-gate-swift15.sh`, `scripts/kld-then-calib.sh`, `scripts/calib-push.py`
|
||
`scripts/make-calib-corpus.py`, `scripts/derive-protected-experts.py`
|
||
`scripts/apply-protected-cast.sh`, `scripts/verify-swift15-ple.py`.
|
||
- Probes: `scripts/probe-frank-archon-kv.py` (reads slots/window/dtype from the deployed env so
|
||
it cannot drift; leg D = decode), `scripts/frank-archon-expert-activity.py`
|
||
`scripts/autoresearch-frank-archon.sh` + `scripts/autoresearch/frank-archon-measure.py`.
|
||
- Deploy and tests: `scripts/deploy.sh` / `scripts/deploy-remote.sh` (`deploy_llm_archon`)
|
||
`tests/frank_archon_unit.bats` (the seams gating the KV-pool arithmetic, the VRAM pair and the
|
||
ttm gate), `tests/changed_tests.bats`, `tests/deployment_scoping.bats`.
|
||
- Retired-era leftovers in-tree: `scripts/patches/glm53-*.patch`
|
||
`scripts/patches/gpu_worker-kv-cap.patch` (fork archaeology; none applies byte-exact today).
|
||
|
||
Related: [[frank-dragoon]] (the abliterated family), [[frank-zealot]],
|
||
---
|