docs: one fact, one owner — endpoint pages own every fleet fact #23

Merged
dank merged 6 commits from wt/20261017-181502 into main 2026-10-19 01:15:56 -07:00
Showing only changes of commit 61011b0a81 - Show all commits
+225 -650
View File
@@ -23,15 +23,17 @@ merge absorbed the former `vllmScripts` repo, history preserved).
## Fleet topology (ACTIVE)
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | **Publisher payload** (swap 2026-10-15, MODEL_FILE-only): `StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe` → **`qwen3.8-next-flash-fp8-iq4r-moe.rad`** (121,969,901,568 B = **116,322 MiB**, Hub-LFS sha256 `0af5e962…` verified at fetch by `scripts/download-frank-archon-iq4r.sh`) — the OFFICIAL Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's author: routed experts as **4-bit codes into a 16-level non-uniform codebook** (128-pt Walsh–Hadamard rotation, GPTQ on a 10M-token calibration + 3 coordinate-descent sweeps, E4M3 scales per 64 weights), **10 whole bf16-protected experts/layer**, int8 W8A8 trunk/lm_head, fp8 HC mixtures, fp8 fixed-scale PLE n-gram (47.7 GiB), **MTP head + bf16 vision tower in the container**; card KL 0.0789 mean all-tokens / top-1 92.85 % vs BF16. **Base-alignment (censored) model** — the abliterated family lives on frank-dragoon. Shape carried VERBATIM from the 2026-10-13 record: **3 × 200,000 fp8-KV** (`--max-num-seqs 3`), **25,600 MiB slab + 5,120 MiB pool = 30,720 of the 30,969 MiB claimable**, 44 GiB/rank pinned host pool, PLE pinned to RAM, MTP depth 2, MNBT 2048, Froggeric v22.5 template. Decode/prefill ladders in the runbook are the **w4dn8 record** (the bar to beat; new-payload probes pending). Rollback: re-download the w4dn8 `.rad` from the HF mirror `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` (LFS sha256 `09dd9306…`) + revert the swap commit — the local copy was **purged 2026-10-17** (it was the last non-iq4r model on the host; the GLM-4.7 GGUF, the radiance kvcache tiers, and every stale compile cache went in the same purge, 763 GiB free). The fp8-native `.rad` + the raw Swift1.5 BF16 source were DELETED from disk 2026-10-15 (Frank's one-time rm, volume for this payload; rebuildable via the fetch/build scripts). | 7770 |
| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-abliterated.hgn`** (66,687,678,432 B ≈ 62.1 GiB resident, Hub LFS sha256 `b5bd925f…` verified at fetch — the **NATIVE 2026-10-11 v2 conversion** of orcarouter's pinned BF16, abliterated, NOT the writer transplant that carried this filename; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea) **+ vision tower** (897,916,416 B, @e053f488) — both **hardlinked** from the swift15 payload dir (the release publishes only the main file; the engine's own `niah` 18/18 + `ppl` 3.968 with mean KL 0.155 vs the swift15 logits certified the stock companions), **vision ON** (`image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset), **TWO slots over ONE `HALOGEN_KV_POOL_POSITIONS=401408` pool (= 2 × ctx) at `HALOGEN_CTX=200704`** (pool RAISED from 200,704 and DEPLOYED 2026-10-18 so BOTH slots can hold a maximum-length request at once; the parity play was frank-zealot's then-401,408 total in-flight context — zealot moved to 786,432 the same evening and dragoon's pool was NOT raised again, because every position is page cache the n-gram table loses; **11.0 GiB reserved, measured on that deploy** (the preflight formula bills 11.3) where 200,704 cost 5.5, 262,144 cost 7.2 and the 2026-10-06 pool measured 14.6; the ceiling does NOT move with it — 262,144 is the native wall and the engine refuses more without `HALOGEN_ROPE_YARN`), harness window 155,648 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (peonist engine recipe — Quat3rnion publishes none; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold), **n-gram table disk-backed** (mmap page cache, `Pin::None`); measured vs the swift15 boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18, PPL +2.3 %, thinking tokens ×1.9|7770|
| Dankclaw | `frank-zealot` | **llama.cpp PrismML fork build** (vendor release `0.2.0-dev` build **10777**, commit `e3de71522`; root systemd unit, relinked — the ONLY loader for this payload; the b10776 floor that gates MTP beside a quantized V cache is moot under exact KV, but the custom tensor type alone keeps the build pinned) | `prism-ml/Ternary-Bonsai-2-9B-gguf` **PQ2_0 carrying the MTP head in the file** (2,675,410,720 B, Hub-LFS sha256 `c79fcdfa…` verified at fetch by `scripts/download-frank-zealot-bonsai9.sh`; `qwen35` hybrid: `block_count 33` = **32 trunk layers** (8 full-attention + 24 Gated DeltaNet at `full_attention_interval 4`) + the `blk.32` MTP block, 4 KV heads × 256 dim, native 262,144; `general.file_type 141` with **202 custom-type PQ2_0 tensors of 442** + `prism.hadamard.*` dequant metadata, which is what locks the runtime) + the publisher's own **Q8_0 mmproj** (624,229,664 B, sha256 `e0cb6f69…`; **Q8_0 is the publisher's requirement** for concurrent image requests — the BF16 tower can crash the server, and this endpoint runs several slots) for **vision**, on the RTX 3090, **TWO 262,144-token slots** (`--ctx-size 524288 --parallel 2` = every conversation STILL at the trained window — the per-slot contract never moved) over **unquantized f16 K + f16 V** (32 KiB/token over the 8 full layers = **16,384 MiB** pool; the POOL halved because exact KV at the old 786,432 pool is 24,576 MiB against the 24,123 MiB the driver leaves addressable — the repo's older "26,112 MiB" figure had doubled the q8_0 number including its 1/32 scale overhead; +100.5 MiB of recurrent state for the two DeltaNet cells, ~50.25 MiB a cell), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF — measured, and now doubly so**: the draft context bills its own f16 KV over the WHOLE pool (3,072 MiB at the old pool, 2,048 MiB at this one) and at ubatch 2048 that squeeze halves decode (167.2 → 109.2 t/s); the documented q8_0-era MTP arm is `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` → 213.5 t/s. Last MEASURED boot (the 3 × 262,144 q8_0 arm this replaces): 17,922 used / 6,203 free, **168.7 t/s** single-stream greedy decode (three runs, 168.4–168.7), **~300 t/s** aggregate across three concurrent streams (102.8 each), 2,877 t/s prefill at 106,418 — the f16 boot's ledger and rates land in the runbook once measured. Cutover 2026-10-18 from Ternary-Bonsai-2-**27B** PQ2_0 at 2 × 200,704 (payload + build 10754 tree on disk as rollback). | 7771 |
| Target | Service | Runtime | Payload | Port |
|--------|---------|---------|---------|------|
| Dankserver | `frank-archon` | radiance, digest-pinned, root unit, TP=2 | publisher `qwen3.8-next-flash-fp8-iq4r-moe.rad` | 7770 |
| Dankclaw | `frank-dragoon` | halogen-flash-server 0.16.2, digest-pinned, root unit | `qwen38-flash-next-v2-abliterated.hgn` + n-gram + vision tower | 7770 |
| Dankclaw | `frank-zealot` | llama.cpp PrismML fork build 10777, root unit — the only loader for this payload | Ternary-Bonsai-2-9B PQ2_0 + Q8_0 mmproj | 7771 |
Retired endpoints (`frank-npu`, and every prior runtime) are in
the [runbook index](#runbook-index). The full swap history lives in
`serverAppConfigs-docs/LLM/index.md` — do not grow it here.
Payload identity, shape, eras, laws and rollback live in ONE page each:
[[frank-archon]], [[frank-dragoon]], [[frank-zealot]]. `scripts/<service>.env`
is the authority for every number; a page that disagrees with it is the bug.
A retired payload is a row in that page's **Eras** table — there is no page
per attempt and no separate swap-history file.
## Repository Layout
@@ -246,71 +248,102 @@ Never `rm`. Move deletions into trash so they stay recoverable:
`git * --no-verify` is forbidden. The trash is a temporary hold, not an
archive.
## Benchmarking Policy (MANDATORY)
## Model Swaps (fleet law, 2026-10-18)
**STOP the production service BEFORE benchmarking** — these servers serve
real users, and a bench beside a live service starves the GPU and corrupts
results. Use the runtime's own harness: `llama-bench` for the llama.cpp
endpoints; a live llama.cpp server (frank-zealot) also reports honest
per-request numbers in its `/completion` `timings` object (prefill/decode
rates, `draft_n`, `draft_n_accepted`), which is how frank-zealot was measured.
halogen has NO `/completion` path at all — frank-dragoon's rates come from
wall-clock around `/v1/chat/completions` plus the boot ledger. Rolling your own
bench requires asking first.
Scratch ports are not used on LLM hosts — a
test process cannot share the single GPU anyway; validate against the service
itself.
**Swap the model. Do not instrument it.** A payload change is: edit the trio
(env / serve script / unit) → `scripts/deploy.sh <host> --service <svc>` →
`/health` answers → one chat turn, one tool call, and one vision probe on an
endpoint that serves images. That is the entire procedure.
(A gufo-era law; frank-dragoon ran ROCmFPX llama.cpp 2026-10-02 → 10-05, the
halogen 27B container 2026-10-05 → 10-06, and is flash-server since, but the
rollback tags keep it live.) **`gufo bench` cannot rank speculative-decode
settings, and will invert the ranking.**
Measured 2026-09-28 on frank-dragoon at depth 32768: its corpus
reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across
draft widths, i.e. the same highly-predictable text every run, and 93 % of
every proposal accepted. It reported 29–37 t/s decode where real traffic
gives ~15 t/s, and it made `--draft-tokens 16` look 6 % faster than 7 when
real text made it 14 % *slower*. Use it for shape, not for A/B. For an A/B,
replay **real** text through the live server and read the `Server-Timing`
header — but note the unit rides in the **name** half (`ttft;dur=402.485`),
so splitting on `=` alone keys the dict `ttft;dur` and every lookup silently
returns nothing. **That header is a llama.cpp feature**: halogen flash-server
0.16.2 answers `/v1/chat/completions` with only `date`, `server` and
`content-*` (verified 2026-10-18), and `/completion`, `/ping`, `/props` and
`/tokenize` are all 404 on it — on frank-dragoon the A/B instrument is
wall-clock around the POST plus `usage.prompt_tokens`. Generate **at least 128
tokens** for the measurement: at 8 tokens, `inter_token` is first-token noise
and the run-to-run spread swamps every real difference.
- **No benchmarking unless Frank orders it.** No pre-swap baseline battery,
no A/B arms, no decode/prefill ladders, no acceptance tables, no numbers
written into any page afterwards. Data exists only when he asks for it,
and then only in the file he names.
- **It doesn't work → roll back.** Revert the commit, redeploy, done. No
post-mortem page, no investigation log, no trial-drop-in forensic tour. If
the fault is not obvious, one line in the endpoint page's *Laws* section is
the whole record.
- What still gates a change: `./tests/run.sh` before commit, the deployer's
per-service health gate, and the Playwright e2e gate. Those are the fleet's
evidence. A bench is not evidence of correctness and buys nothing a swap
needs.
This covers **any GPU experiment, not just benchmarks** — including a bare
2-rank RCCL/`torch.distributed` probe. On 2026-09-16 one such probe was run
against the live service "because it only needs 200 MB of VRAM"; the host went
hard down ~20 minutes later (no ARP reply, 100% ICMP loss from two subnets, no
recovery — a power cycle was required). A hung AQL dispatch wedging the GPU
driver is the plausible mechanism, and this host has precedent for going
unreachable until a power cycle. If an experiment needs GPUs, stop and mask
first, or use a host that serves nobody. Telling an off machine from a wedged
one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a
merely wedged kernel usually still answers ARP.
### When a bench IS ordered
**STOP the production service first** — these servers serve real users, and a
bench beside a live service starves the GPU and corrupts the result. That
covers **any GPU experiment, not just benchmarks**, including a bare
2-rank RCCL/`torch.distributed` probe: on 2026-09-16 one was run against a
live service "because it only needs 200 MB of VRAM" and the host went hard
down ~20 minutes later (no ARP reply, 100 % ICMP loss, power cycle required;
a hung AQL dispatch wedging the driver is the plausible mechanism). Telling
an off machine from a wedged one: `ip neigh show <ip>` from a peer — FAILED
means off or NIC dead; a merely wedged kernel usually still answers ARP.
1. Stop and mask (self-healing would restart it):
- Dankclaw `frank-dragoon` (root unit, halogen podman on the iGPU):
`sudo systemctl stop frank-dragoon && sudo systemctl mask frank-dragoon`
- Dankclaw `frank-zealot` (root unit, llama.cpp on the 3090):
`sudo systemctl stop frank-zealot && sudo systemctl mask frank-zealot`
- Dankserver `frank-archon` (root unit, radiance on both R9700s, TP=2):
`sudo systemctl stop frank-archon && sudo systemctl mask frank-archon`
2. Bench.
NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in
`/etc/systemd/system/` is itself a symlink into the checkout — mask
refuses to replace it). Stop is enough for the bench window; the deploy
restores the unit, so re-deploy after benching instead of unmasking.
3. Unmask and start; verify `curl http://192.168.0.<host>:7770/health`.
`sudo systemctl stop frank-dragoon frank-zealot` on Dankclaw,
`sudo systemctl stop frank-archon` on Dankserver, then `mask` each.
`mask` FAILS on a relinked unit (the file in `/etc/systemd/system/` is
itself a symlink into the checkout); stop alone is enough for the window,
and the deploy restores the unit — re-deploy after benching rather than
unmasking.
2. Bench against the service itself. No scratch ports on an LLM host — one
process owns the card. Use the runtime's own harness (`llama-bench`; a
live llama.cpp server also reports honest `timings` per request, including
`draft_n`/`draft_n_accepted` — but on frank-zealot's PrismML fork build
10777 that object only reaches you on **`/v1/chat/completions`**: a greedy
`ignore_eos` run on the raw `/completion` path there answers **500 "The
model produced output that does not match the expected Content-only
format"** (measured 2026-10-18). halogen has no `/completion` path at all,
so its rates are wall-clock around `/v1/chat/completions` plus the boot
ledger. Generate ≥128 tokens: at 8, `inter_token` is first-token noise and
run-to-run spread swamps every real difference.
3. Unmask, start, verify `curl http://192.168.0.<host>:7770/health`.
Retire every bench drop-in the same session
(`systemctl show <svc> -p DropInPaths` must come back empty) and write the
numbers into the file Frank named — `/tmp` and the journal are not records.
Instrument traps that survive from the era when this was routine:
`gufo bench` cannot rank speculative decode and **inverts the ranking** (its
corpus reports `acceptance=0.933 ± 0.000` with an identical trace sha256
across draft widths — the same predictable text every run); llama.cpp's
`Server-Timing` carries the unit in the **name** half (`ttft;dur=402.485`),
so splitting on `=` keys the dict `ttft;dur` and every lookup silently
returns nothing; halogen 0.16.2 answers `/completion`, `/ping`, `/props` and
`/tokenize` with 404 and sends no `Server-Timing`; and the first request
after a boot is a warmup artifact (measured 2.6 t/s prefill then 1,258 t/s on
the next), so never report request #1.
**CRITICAL:** never `systemctl disable` a system unit on these hosts — it
deletes the unit file from `/etc/systemd/system/`, not just the symlink.
`mask` symlinks to `/dev/null` and is safe/reversible.
## Documentation Policy (fleet law, 2026-10-18)
**One fact, one owner.** A page owns exactly one thing — an endpoint, a
service, or a technique — and every other file links to it with `[[Page]]`.
Restating a shape, a digest, a port list, or a measured number in a second
file is a defect: it is the next thing to go stale.
- **One page per endpoint, forever:** `serverAppConfigs-docs/LLM/<endpoint>.md`
— current shape, an **Eras** table (`window | payload | runtime | shape |
replaced because | rollback`), the **Laws** the endpoint taught, the wire
contract, and the files that own it. A payload swap edits the shape block
and appends one Era row. It never creates a page, and the retired era's
page goes to `.AGENTTRASH/` once its rollback path and its laws are in the
table.
- **`log.md` records incidents and laws** — anything with a causal story
(why the host went down, why a knob resets at reboot, why the RAM was the
fault). A model swap is not an incident: it is an Era row and nothing else.
- **`AGENTS.md` carries operator-critical numbers only** — port, context
ceiling, slot shape, unit/env/serve trio — plus the cross-cutting laws.
- **Index pages are locators:** one row, one clause, no numbers.
- **A measured number is written once:** in the env-file header beside the
serve script that consumes it, and in the endpoint page. Every other page
cites it by link.
- Technique pages (spec decode, KV math, offload budgets, PP bandwidth) stay
technique pages; they stop carrying per-endpoint throughput tables.
## Model Config (harness)
The harness reaches the fleet through `.omp/agent/models.yml` in the
@@ -366,424 +399,71 @@ endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's
audio encoder, and left again with the Gemma4 retirement on 2026-10-01
(neither the MiMo nor the Qwen3.5 tower carries audio).
### frank-dragoon — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 NATIVE abliterated (Dankclaw, ACTIVE)
Each endpoint has ONE page in the vault: shape, Eras, Laws, wire contract,
rollback. The numbers below are only what a consumer or a deploy gate needs;
the env file beside the serve script is the authority, and a page that
disagrees with it is the bug.
The **flash-server engine** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`,
digest-pinned, published tag 0.16.2, 3.6 GB image) as a podman container
driven by `scripts/serve-frank-dragoon.sh` under a **root systemd unit**
(`scripts/frank-dragoon.service`, `User=dank`, `Restart=always`,
`ExecStop=podman stop -t 60 frank-dragoon-halogen`, the script runs `podman
run` FOREGROUND — the container's exit code is the service's exit so
`Restart=always` stays real, and the engine ledger lands on disk) on port
7770 from the Strix Halo iGPU. Engine stdout goes to
`/home/dank/logs/frank_dragoon.log` — **the boot ledger (weights resident, KV
pool, MTP head, prompt cache, any clamp) lives there, not in the journal**.
Cutover 2026-10-17 from the Swift1.5-finetune shape
([[frank-dragoon-flash-server-v2-swift15]], retired the same window; it
itself cut over the halogen 27B-dense era on 2026-10-06).
### frank-dragoon — halogen flash-server, Dankclaw iGPU, :7770
Serving `Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated`:
`qwen38-flash-next-v2-abliterated.hgn` (66,687,678,432 B ≈ 62.1 GiB, sha256
`b5bd925f…` — the **NATIVE 2026-10-11 v2 conversion** of orcarouter's pinned
BF16 checkpoint, abliterated; NOT the writer transplant that carried this
filename, which measured 66,687,678,464 B — 32 bytes apart, same filename
family, different model) + the **REQUIRED** n-gram lookup table
`qwen38-flash-next-ngram.hgn` (51,200,246,144 B, peonist @5cc17cea) + the
vision tower `qwen38-flash-next-vision.hgn` (897,916,416 B, @e053f488) + the
flat `tokenizer/` directory (6 files) — the three companions **hardlinked**
out of the swift15 payload dir, because the release publishes only the main
file and no source-matched pair exists. Cutover 2026-10-17 from the Swift1.5
finetune, chosen by Frank for the base checkpoint itself plus the card's
memory-saving pool.
- Owns: `scripts/frank-dragoon.env`, `scripts/serve-frank-dragoon.sh`,
`scripts/frank-dragoon.service`, container `frank-dragoon-halogen`, image
digest-pinned `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`.
Engine stdout (the boot ledger: weights, KV pool, clamp) goes to
`/home/dank/logs/frank_dragoon.log`, NOT the journal.
- Shape: `HALOGEN_CTX=200704` is the **per-request ceiling** (a longer prompt
is a 400, never a truncation); `HALOGEN_KV_SLOTS=2` over ONE
`HALOGEN_KV_POOL_POSITIONS=401408` pool — pool positions buy concurrency,
the per-request window never moves; `HALOGEN_SPEC_ADAPT=0` keeps the MTP
drafter always armed (the adaptive default disarms it and halves decode);
server sampler 1.0/0.95/20.
- Harness contract: `models.yml` `contextWindow: 155648` = ceiling − 45,056.
The clamp is `min(45056, window − est − 4096)` and the chars/4 estimate
under-counts, so a harness window equal to the ceiling makes every long
request a guaranteed 400. Reasoning arrives in `reasoning_content`; the
template rejects `reasoning_effort` outside `xhigh|medium|low`; images are
OpenAI `image_url` parts (probe ≥256 px); `response_format` unsupported;
`/completion`, `/ping`, `/props`, `/tokenize` are 404.
- Page: [[frank-dragoon]].
**The format is the lock.** The v2 `.hgn` loads ONLY in the
flash-server line — the 27B-era halogen 0.1.4 container refuses it, so the
engine moved with the model. The MTP head is INSIDE the checkpoint
(`HALOGEN_MTP_HEAD` stays unset; the separate `qwen38-flash-next-mtp.hgn`
sidecar is deliberately not fetched).
### frank-zealot — llama.cpp (PrismML fork), RTX 3090, :7771
**Product facts vs the 27B era, both regressions FIXED:** the base is
**abliterated** (the 27B was a censored-base finetune; refusals were an
accepted trade there) and **vision is ON** (the 27B engine silently
ignored its encoder — frank-zealot owned images for exactly one day at this
endpoint).
- Owns: `scripts/frank-zealot.env`, `scripts/serve-frank-zealot.sh`,
`scripts/frank-zealot.service`; fork build `llama-prism-b10777` is the ONLY
loader (the payload carries the fork's custom PQ2_0 tensor type plus
`prism.hadamard.*` dequant metadata — stock llama.cpp refuses it, and a
loader ignoring that metadata dequantizes wrong); payload
`Bonsai2-9B-MTP/…PQ2_0.gguf` + the publisher's **Q8_0** mmproj (the BF16
tower can crash the server with concurrent image requests).
- Shape: `--ctx-size 524288 --parallel 2` = **two conversations at the
model's native 262,144**, `CACHE_TYPE_K/V=f16`, batch 4096 / ubatch 2048,
`SPEC_TYPE=none`, `--flash-attn on`, `--jinja`, `--metrics`.
- Harness/gate contract: `contextWindow: 262144` is the **per-slot** window
(`n_ctx_slot = CTX_SIZE / PARALLEL`), and llama.cpp answers a longer prompt
with a 400 rather than truncating. `--metrics` is mandatory or Prometheus
reads 501 = DOWN beside a healthy service. Presence penalty has no GGUF
metadata key, so it must reach argv. This endpoint is the fleet's
compaction hub.
- Page: [[frank-zealot]].
**Engine laws that shape the config** (all in `scripts/frank-dragoon.env`
with evidence): `HALOGEN_SPEC_ADAPT=0` is MANDATORY — the default adaptive
policy disables the MTP head for whole requests after low acceptance and
HALVES decode; the head stays on and temp-1.0 traffic pays some acceptance
for it (accepted). `HALOGEN_CTX=200704` is the PER-REQUEST hard ceiling (400
naming the limit, never truncation) — the checkpoint's native window is
262,144 and was served until 2026-10-17, when Frank lowered it for RAM. The
**400-zone law is invariant to the value**: the harness clamps max_tokens to
`min(45056, contextWindow − est − 4096)`, its chars/4 est UNDER-counts
the rendered prompt by ~3 % (measured: est 152,938 → server 157,388,
request 43,158 vs room 42,804), so a harness window == engine ceiling
makes every long request a guaranteed 400. `models.yml`
`contextWindow: 155648` (= ceiling − 45056) keeps that gap (worst-case
request ≈ 156.2k ≤ 200,704); compaction fires at 85 % of the harness window
= 132,301 est ≈ 136k measured (the 180k target died with the 262,144 window).
`HALOGEN_KV_SLOTS=2` is LEGAL here — the 27B "slots>1 disables the drafter"
law does NOT carry over; batching and speculation coexist (the card's
quickstart ships 2 slots with the head), so 2 is concurrent capacity.
`HALOGEN_KV_POOL_POSITIONS=401408` — **2 × ctx, RAISED from 200,704 on 2026-10-18**
(Frank: dragoon's total in-flight context was HALF frank-zealot's 401,408
pool — doubling the POOL is what fixes that, not doubling the window) — is
**ONE pool behind both slots**, ledger form `2 slots over ONE <pool>-position KV
pool, each request up to HALOGEN_CTX`, and at pool == slots × ctx two
MAXIMUM-length conversations DO coexist. The PER-REQUEST ceiling stays
`HALOGEN_CTX=200704`: the checkpoint's native window is 262,144 and the engine's
own 400 text refuses to exceed it without `HALOGEN_ROPE_YARN`, so "more context"
bought here is CONCURRENCY, not a longer single conversation. ~28 KiB/position ⇒
the 2026-10-18 deploy RESERVES **11.0 GiB** (`reserving the KV pool, 401408
positions (about 11.0 GiB)`; the preflight formula bills 11.3 with block scratch
+ ~115 MiB/slot of O(1) state), where 200,704 cost 5.5, 262,144 cost 7.2 and
the 2026-10-06 shape's 524,288 measured 14.6 — quote the reservation.
`HALOGEN_MAX_TOK=8192` is only the per-request DEFAULT (the
harness sends at most its `maxTokens` 45056); `HALOGEN_MAX_TOKENS_CAP=65536` +
`HALOGEN_QUEUE_TIMEOUT=7200` are a coupled pair (FLAGS.md table).
**A server-side sampler EXISTS** (`HALOGEN_TEMPERATURE/TOP_P/TOP_K` =
1.0/0.95/20, the card's thinking recipe) — an absent temperature is no
longer greedy; `models.yml` extraBody carries the same values, consistent
not load-bearing. Prompt cache is ON by default (`PROMPT_CACHE=2`): a warm
100k follow-up costs ~2 s vs ~88 s cold — the 27B engine's manual
`HALOGEN_CACHE_MB` pin era is over. `response_format` is unsupported.
Thinking lands in **`reasoning_content`**; the v2 chat template rejects
`reasoning_effort` outside `xhigh|medium|low`, so the `models.yml` fold
stays mandatory. Thinking tokens COUNT against the budget (empty content +
`finish_reason: "length"` when exhausted mid-thought). Vision: OpenAI
`image_url` content parts only; ≥256 px probe images or the model answers
"video".
### frank-archon — radiance, two R9700s TP=2, Dankserver :7770
**Memory ledger** (121.2 GiB unified; the referee is the boot ledger): weights
62.9 (trunk 62.11 + vision 0.84) + KV **11.0** + working 5.7
(`3.8 + 8.4 × MAX_TOK/16384`) = **79.6 GiB in all** (boot ledger, the
401,408-position deploy of 2026-10-18), the **1.3 GiB** prompt cache (12 entries
× 111 MiB at ctx 200704) booked separately by the ledger, + `HALOGEN_HOST_
RESERVE_GIB=20` → ~20 GiB margin over baseline co-tenancy (`MemAvailable` read
22 GiB right after the raise; the pool-equals-ctx boot printed 74.1 GiB with
~30 GiB of margin). The 51.2 GB n-gram table rides as mmap'd page cache beside
the checkpoint (ledger: `mapped 51.2 GB, NOT pinned (Pin::None)`) — which is why
every pool position is a double cost: **RAM the pool takes is RAM that table
loses**, so the warm prompt-cache ratio is re-measured after a raise.
`HALOGEN_FLASH_PIN_TRUNK=0` is REJECTED by the v2 loader, so disk-backing the
trunk is not a lever on this format. READ the boot ledger and re-size if the
engine prints a clamp.
Fetch: `scripts/download-frank-dragoon-v2abl.sh` into
`/home/dank/models/halogen-v2-abliterated/` (`-f` + exact byte counts + LFS
sha256 + `--verify-only`, no token on the command line; resume goes against
the RESOLVED CDN URL because curl drops `Range` across HF's 302) — the main
file is fetched, the three companions are HARDLINKED from the verified
swift15 dir (a hardlink is a regular file, so the no-symlink serve gate
passes without a 47.7 GiB copy). The serve gates mirror the deployer EXACTLY
(byte-exact checkpoint + n-gram + vision tower, flat tokenizer with all six
files as regular files) — the 2026-10-01 crash-loop lesson.
Runbook:
`serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md`.
Rollback: revert the swap commit and redeploy — the Swift1.5 payload
(`/home/dank/models/halogen-flashnext-v2/`, re-verified by
`scripts/download-frank-dragoon-flashnext.sh`), the 27B payload
(`/home/dank/models/halogen-qwen3.8-27b/`) and the halogen 0.1.4 image
digest all remain on disk; the ROCmFPX door behind that
(`scripts/build-rocmfpx.sh`, pin `36e9acd`) is untouched.
### frank-zealot — llama.cpp (PrismML fork) on the RTX 3090 (Dankclaw, ACTIVE)
**The PrismML fork build of llama.cpp** (vendor release, `--version` =
`0.2.0-dev (build 10777, commit e3de71522)`;
`/home/dank/GitPackages/llama-prism-b10777/llama-server`) as a **root systemd
unit** (`scripts/frank-zealot.service`, `relink_unit`ed into
`/etc/systemd/system/`), serving `prism-ml/Ternary-Bonsai-2-9B-gguf` at
**PQ2_0 with the MTP head IN the file** (2,675,410,720 B, Hub-LFS sha256
`c79fcdfa…` verified at fetch) **plus the publisher's own Q8_0 mmproj**
(624,229,664 B, sha256 `e0cb6f69…`), both `/home/dank/models/Bonsai2-9B-MTP/`,
on port 7771. Cutover 2026-10-18 from the Bonsai-2-**27B** PQ2_0 era
(2 × 200,704, same card, same fork family).
**The runtime is part of the payload identity — twice over.** The file is
`general.file_type 141` — **202 of the 442 tensors carry the fork's PQ2_0 type**
(the rest are f32 norms/scalars plus 8 q8_0 in the MTP block) — beside
`prism.hadamard.*` metadata (normalized Sylvester–Walsh transform): stock
llama.cpp refuses the custom type at load, and a loader that
ignored the Hadamard metadata would dequant WRONG — same lock class as the
ROCmFPX era. On top of that lock, **the fork gates MTP beside a quantized V
cache on build `prism-b10776-d00e252` or newer**, which is why the prebuilt
moved 10754 → 10777 (exact f16 KV makes that gate moot — the custom type alone
pins the runtime); the 10754 tree stays on disk as the 27B rollback runtime.
Retargeting the payload without the fork ships an endpoint that cannot load,
so the seam pins both paths.
**Architecture: qwen35 hybrid** — read from the GGUF's own header: `block_count`
**33 = 32 trunk layers + the `blk.32` MTP block** (the MTP block has its own
attention + FFN in q8_0, which is why the draft context bills real KV), and
`qwen35.full_attention_interval = 4` ⇒ **8 full-attention + 24 Gated
DeltaNet**, 4 KV heads × 256 dim, native 262,144. **KV accounting:** the pool
scales with the 8 full layers — 8 × 4 × 256 = 8,192 elements/token per cache,
so **the deployed exact pair f16 K + f16 V bills 32,768 B/token = 32 KiB** =
**16,384 MiB** at the 524,288 pool. The q8_0 pair this endpoint shipped first
billed **17,408 B/token** (1.0625 B/element: 32 codes plus one 2-byte scale per
32-block) = **13,056.00 MiB** at the old 786,432 pool, ledger-exact — exactly
HALF the 27B's 34,816 B/token. **Exact KV at that 786,432 pool is 24,576 MiB,
more than the 24,123 MiB the driver leaves addressable, so the POOL is what
pays for f16 — the window never moved.** (The "26,112 MiB" figure this repo
carried doubled the q8_0 number *including* its scale overhead; the true ratio
is 1.882×, not 2×.) The 24 DeltaNet layers bill a fixed recurrent state per
SLOT — ~50.25 MiB a cell, so the 3-slot ledger printed `RS buffer size =
150.75 MiB (3 cells, 32 layers, 3 seqs)` and two slots bill 100.50, while the
SPEC-ARMED arms print 452.25 MiB (MTP depth 2 bills three seqs per slot, so 9
cells) — so pool arithmetic alone under-prices the shape.
**The shape (2026-10-18, resliced the same evening):** `--ctx-size 524288
--parallel 2` = TWO conversations of 262,144, every one still at the model's
TRAINED window (the first arm was THREE — Frank: "if you can possibly fit a
third context, do it" — and the exact-KV reslice handed one back),
`--flash-attn on`, **unquantized f16 K + f16 V**, batch 4096 / **ubatch 2048**
(the measured CUDA optimum carried from the MiMo/Swift A/Bs on this card; it
also sets the compute-buffer size, which is what decides whether MTP fits),
`--log-verbosity 4`, `--metrics`, `--jinja`, `--mmproj` for vision,
**speculative decoding OFF** (`SPEC_TYPE=none`). The MEASURED record still
belongs to the arm this replaced (3 × 262,144, q8_0 K + q8_0 V): boot **17,876
MiB used / 6,249 free**; compute 1,409 MiB CUDA + 1,201 MiB CUDA_Host, projector
595 + 248 MiB; **167.2 t/s** greedy decode, 158.9 t/s single-stream chat,
**280.2 t/s aggregate across three concurrent streams**, **2,877 t/s prefill at
106,418** tokens — the 27B record (76.6 decode / 109.4 aggregate / 691.7 prefill
at 166k) beaten on every axis. The exact-KV arm's ledger and rates go in the
runbook once booted; its projection is ~21,200 MiB used / ~2,900 free, since
only the KV pool (+3,328) and one recurrent-state cell (−50) move. The **first
request after boot is a warmup artifact** (37.7 t/s class) — never report it.
**Why MTP is off even though the head is in the file.** The drafter is GOOD —
acceptance 0.798 / 0.617 per position, mean accepted length 2.41 — and the card
recommends depth 2. It still loses because the draft context bills its OWN
**f16** KV over the WHOLE
pool (4,096 B/position — **3,072.00 MiB** at the 786,432 pool, 4,830 MiB with
its compute buffers; **2,048.00 MiB** at the shipped 524,288 pool) and ubatch
2048 refuses to give it up. Booted arms, same card, same traffic (2026-10-18,
every row q8_0 K + q8_0 V at the 786,432 pool):
| slots | spec | ubatch | used / free | decode |
|---|---|---|---|---|
| 3 × 262,144 | off | 2048 | 17,876 / 6,249 | **167.2** |
| 3 × 262,144 | on | 2048 | 22,706 / 1,419 | 109.2 |
| 3 × 262,144 | on | 1024 | 21,400 / 2,725 | **213.5** |
| 2 × 262,144 | on | 2048 | 17,178 / 6,947 | 213.2 |
| 1 × 262,144 | on | 2048 | 11,652 / 12,473 | 214.2 |
**MTP is worth +28 % decode whenever the card has room**; the squeeze, not the
drafter, is what halves it at three windows. The arm that WOULD run it at three
windows is `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` (213.5 t/s single-stream,
250.8 three-stream aggregate) and pays prefill −17 % and 2.7 GiB of headroom on
a card whose projector worst-case lands AFTER the engine's own fit check. Under
**exact KV the drafter is arithmetically out**: the trunk projects ~21,200 MiB
of the 24,123 MiB addressable before the draft context's 2,048 MiB and its
buffers are even asked for, so the doors back to speculation are ONE window per
conversation, or KV back to q8_0.
The compaction-hub load is what makes exact KV affordable: prefill is
weight-bandwidth bound and does not re-read the cache per generated token, so
the dtype is paid in decode and concurrency (two conversations now, not three),
not in the work this endpoint exists to do. `SPEC_TYPE=none` strips the WHOLE
spec set (`--spec-type` and every `--spec-draft-*`) — flags move as a set or the
arm measures a crash instead of an absence.
**Sampler**: the recipe lives in the checkpoint (`general.sampling.*` =
temperature 1.0 / top_p 0.95 / top_k 20 / min_p 0 / repeat_penalty 1.0,
confirmed in the engine's own `/props`) and llama.cpp reads it itself.
**Presence penalty is the one recipe value with no GGUF metadata key**, so
`PRESENCE_PENALTY=1.0` must reach the argv; `""` is the documented opt-out, and
the serve script therefore uses `${PRESENCE_PENALTY=1.0}` (NOT `:=`, which
would silently re-add the flag on an explicit empty value).
**Vision**: the publisher's own **Q8_0** mmproj via `--mmproj` — Q8_0 is
MANDATORY, not a preference: the card says the BF16 tower "can crash
llama-server when several image requests run at once", and `--parallel 2` is
exactly that shape. Probe traps carry over: ≥256 px probe images or the model
answers "video", only the OpenAI `image_url` content part reaches the model,
and thinking tokens count against `max_tokens` — budget ≥512 or the probe
returns EMPTY `content` with `finish_reason: "length"` and the endpoint looks
blind while it is only mid-thought.
**Base alignment**: Bonsai-2 is the censored BASE lineage. Unlike the 27B, the
CRACK re-export on disk (`/home/dank/models/Bonsai2-CRACK/`) is byte-identical
geometry to the 27B it matched, NOT to this 9B file — if a refusal ever
surfaces there is no one-line `MODEL_FILE` swap; a 9B dealigned export would
have to be fetched first.
**Compaction hub**: `models.yml` routes `compactionModel` for
`frank-archon`, `frank-dragoon` and `frank-zealot` (the 3090 endpoint) here —
262,144 (still the PER-SLOT window after the exact-KV reslice) is the widest
window in the fleet and the only one that comfortably fits a worst-case
compaction input (~170k tokens). With `--parallel 2` a compaction request still
runs beside ONE live generation, and with two already busy it waits — the Gemma4
era's `--parallel 1` law at half strength, which is the concurrency price of
exact KV. Retrieval through a
25,960-token prompt was verified exact after the swap.
**Contract law learned at verbosity 4:** a raw `/completion` request prints
`failed to parse reasoning content (… parse_reasoning=true)` whenever the
model closes a bare prompt with `` — cosmetic
(`probe_request.cpp:789`); the `/v1/chat/completions` path parses
`reasoning_content` correctly, which is the only path consumers use. Do not
lower `VERBOSITY` to silence it: the KV/buffer ledger is what VRAM forensics
read. `--metrics` is required or the Prometheus scrape reads 501 = DOWN.
Runbook:
`serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai9-mtp.md`.
Rollback: revert the swap commit — the 27B payload
(`/home/dank/models/Bonsai2-PQ2_0/`) with the 10754 fork tree
(`~/GitPackages/llama-prism/`), the Qwen3.5-9B Q6_K payload
(`/home/dank/models/Qwen35-9B-HauhauCS-Q6K/`) with the master CUDA build
(`~/GitPackages/llama-master/build-cuda/`), the Gemma4 payload
(`/home/dank/models/Gemma4-26B-A4B/`), the MiMo Q6_K payload
(`/home/dank/models/MiMo-Q6K/`), the Swift payload
(`/home/dank/models/Swift15-UncIQ3XXS/`) and the Ornith Q8_0 payload
(`/home/dank/models/Ornith-Q8/`) all remain on disk.
### frank-archon — Radiance (StillDeadcode), publisher Qwen3.8-Flash-Next iq4r, TP=2 (Dankserver, ACTIVE)
The **radiance engine** (StillDeadcode, Apache-2.0; image
`stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned, **pin 1.0.13**
for serve; `libr4d` gfx1201 kernels, plain `podman run`, root
systemd unit, container name = unit name) serving the
**publisher's container of the official Qwen/Qwen3.8-Flash-Next**:
`StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe` →
`qwen3.8-next-flash-fp8-iq4r-moe.rad` (116,322 MiB, Hub-LFS sha256
verified at fetch by `scripts/download-frank-archon-iq4r.sh`; recipe
`qwen4exp-w4nl64-i8-hc8m`): **routed experts as 4-bit codes into a
16-level non-uniform codebook** after a 128-point Walsh–Hadamard rotation
(E4M3 scale per 64 weights; codes by GPTQ on a 10M-token calibration with
per-expert down-proj Hessians, refined by 3 coordinate-descent sweeps),
**10 whole bf16-protected experts/layer**, **int8 W8A8 trunk/lm_head**
(scale per 128 columns searched), **fp8 HC mixtures**, PLE n-gram table at
fp8 fixed scale (47.7 GiB), **MTP head and bf16 vision tower in the
container**. Card quality (full-vocab teacher-forced KL vs bf16, ~75K
positions): mean 0.0789 all-tokens / 0.0239 assistant turns, top-1 92.85 %.
**MODEL_FILE is the only serving argument that moved** (2026-10-15); every
number of the 2026-10-13 shape below carries verbatim. The plugin's
**one-table-one-encoding law** (fleet-measured at rad-convert declare):
one expert table takes ONE encoding; only whole bf16 experts are legal
beside it. Rollback: revert the swap commit AND re-download the in-house
w4dn8 `.rad` (KLD-gated Swift1.5 conversion) — it is **no longer on disk**
(purged 2026-10-17); the byte-exact copy is published as
`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`
(LFS sha256 `09dd9306…`). The fp8-native `.rad` was deleted 2026-10-15
(rebuild: `scripts/download-frank-archon.sh` +
`scripts/build-frank-archon-rad.sh`).
**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 3`,
`--max-model-len 200000` (the harness contextWindow must quote this),
`--placement expert_tiered --host-pool-mib 44928` + **explicit VRAM
partition `--vram-weights-mib 25600 --vram-kv-mib 5120`** (the vram-pair
replaced `--expert-vs-cache-ratio`; the two are ONE budget —
`weights + kv ≤ 30,969 MiB`, derived from the engine's own budget table: card
32,624 MiB, the pair at 29,120 settling 2,058 MiB free against the engine's
209.19 MiB floor (`--gpu-headroom-mib 96` + 113.19 static), so 1,849 MiB is
claimable and the old 29,120 bound was my hand-picked 1 GiB reserve, not a
hardware wall), `--kv-cache-dtype` **`fp8` — quantized KV, restored 2026-10-13
because it is the only dtype that funds the slot count at this window**,
`--num-speculative-tokens 2`, `--max-num-batched-tokens 2048`, prefix cache
8 GiB pinned host + 64 GiB disk at `/home/dank/radiance-state`,
**`--ngram-placement ram`**, Froggeric v22.5 chat template vendored
(`scripts/qwen38-froggeric-v22.5.jinja`, `--override-chat-template`). Boot
ledger (2026-10-13, pair 25600/5120): `kv cache 5.00 GiB … backs 3 sequences
all at 200000 tokens at once, which is the worst case` — `kv_attn` page
26.00 KiB (fp8) at 607,740 tokens (the sizing law predicted 608,027),
`kv_qsa_bkey` 3.25 KiB at the same token count, `kv_gdn_state` 54 MiB × 6 slots
(3 sequences × 2 copies) = 324 MiB; the sampler prints **9 rows**
(3 × (1+2), inside the 25-row `wht6` exactness bound); the slab filled to
563 KiB of slack with **17,548 slots** (16,235 at 24,000, 14,753 at 22,208 —
every 1,792 MiB buys ~1,300 resident experts, and the pinned host pool hands
the same experts back: 8,893 → 7,580 slots); `unclaimed 477.30 MiB`, cards
settled ~336 MiB free against the engine's 209.19 MiB floor, KV lending
4.22 GiB to the expert cache. Prefill tracks this slab only up to a
**threshold**: 22,208 → 3,397 and 24,000 → 3,290 t/s at 43.5k are one band
(±3 %), and the 25,600 arm measured **3,290** — the third step bought nothing
(the 3,584 arm did not reproduce at 3 slots); the cliff is at ~26 GiB
(26,071 → 4,379 on 2026-10-11's arm), so below that the slab is a threshold,
not a slope.
**Decode ladder (2026-10-13, 512 tokens, greedy, this shape)**: 99.0 t/s
single-stream at 43.5k, 93.3 at 197k — decode is nearly window-free (−6 % for
4.5× depth); **3 concurrent streams give 113.9/43.3/10.9 t/s = 18.4 t/s
aggregate**, so the slot count is paid in decode concurrency (a fused-step
lottery), not in prefill; leg D of `probe-frank-archon-kv.py` measures it.
Shape notes carried from the fp8-native era and unchanged: wht6 gains,
MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk
prefix cache, and the rule that a prompt over `--max-model-len` is
refused, never truncated. What changed: the fp8-native era's NVMe expert
streaming (the 3.5 GB/s wall) is gone — the w4 resident set fits the
host pool, so decode ran 122–173 t/s in the campaign arms and ~190 t/s
on the fp8 shape (512-token streamed probe). The campaign also
measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth,
accept 0.7246).
**The KV dtype law (measured 2026-10-11, applied at 4 slots 2026-10-13)**:
`--kv-cache-dtype` takes ONLY `bf16|fp8` — one dtype for the whole cache, so
llama.cpp's exact-K/quantized-V pair (the 9B-era frank-zealot ran it; the
Bonsai-2 swap runs q8_0/q8_0) has NO radiance
equivalent; the rung under bf16 is fp8 on both halves. Measured per rank from
the two boot ledgers: `kv_attn` (13 layers, 4-token blocks) pages 26.00 → 52.00
KiB, while `kv_qsa_bkey` (13 layers) stays 3.25 KiB at BOTH dtypes (the QSA
block key is an index, not attention state) — so a token costs 7.3125 KiB at fp8
and 13.8125 KiB at bf16: **1.89×, not 2×**. Pool sizing must also pay the
dtype-independent terms that live in the same pool (~110 MiB a sequence of
recurrent state + 448 MiB of prefix checkpoints):
`(vram_kv − 110·seqs − 448) MiB / rate` — bf16 at 6,912 gives 462,946 where the
engine reported 462,752 (0.04 % page rounding); fp8 at 6,912 with four slots
gives 843,566, i.e. the deployed capacity contract 4 × 200,000. **That is why
the dtype is fp8: the pool must back `seqs × max-model-len` as a hard limit, and
bf16 at four slots costs 11,679 MiB — the slab would fall to 17,441 MiB (−33 %),
where fp8 leaves it at 22,208.** The cost is paid twice on this engine, and
half-refunded: idle KV VRAM is also the expert cache's spillover, and the bigger
pool *lends* 6.08 GiB to the slab where the 3,543 MiB fp8 pool lent 2.69 GiB.
`--max-num-seqs` rides 4 × (1 + spec) = 12 fused-all-reduce rows, under the
25-row bound where wht6 is bit-exact (the "seqs 2 cannot boot" law belongs to
the retired tcclaviger fork's torch.compile, not radiance). Disk-tier sizing
law: the engine forces ~9.6 GiB of disk-tier weight buffers + arena onto the
weights side, so `weights + kv` is capped near 29.6 GiB/card; boot with the
pair, never `--kv-cache-size-mib`.
Shape notes carried from the fp8-native era and unchanged: wht6 gains,
MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk
prefix cache, and the rule that a prompt over `--max-model-len` is
refused, never truncated. What changed: the fp8-native era's NVMe expert
streaming (the 3.5 GB/s wall) is gone — the w4 resident set fits the
host pool, so decode ran 122–173 t/s in the campaign arms and ~190 t/s
on the fp8 shape (512-token streamed probe). The campaign also
measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth,
accept 0.7246).
**The pinned-RAM ledger is why this is a root unit**: 2 × 44 GiB expert
pools + the 47.7 GiB n-gram copy (`--ngram-placement ram` is pinned RAM
the card reads) + 8 GiB prefix host ≈ **147 GiB** charged against
`RLIMIT_MEMLOCK` — the TTM floor is set just above it.
`ttm.pages_limit`/`pages_max` cap GPU-mappable pinned pages, and **the knob is
runtime state whose default is half of RAM**: `scripts/ttm-preflight.sh` (sourced
by the serve script) therefore RAISES it to `FRANK_ARCHON_TTM_TARGET_MIB=204800`
before refusing, because the 2026-10-12 power failure reset it to 125,988 MiB
and the preflight re-refused 6,919 times. A manual sysfs write in the stop
window is a convenience, not the repair; kernel cmdline persistence
(`ttm.pages_limit=52428800`) is the belt to it.
**Contract facts** (source-verified): the HTTP listener binds AFTER the
full load, so a bare `curl /health` gate is honest (`/health` == `/ping`,
200 serving / 503 stopping); sampling params temperature…preserve_thinking
accepted, **unknown params 400** (models.yml's extraBody set is fully
inside it); `response_format` supported; `logprobs` REFUSED; reasoning in
`reasoning_content` and tool calls in the qwen3.8 dialect via the auto
chat format — no parser flags; vision native (the 27-layer Qwen-VL tower,
`--mm-max-patches`); `/metrics` carries vLLM-compatible names +
`radiance:*` families — the Dankclaw Prometheus scrape is re-added.
Runbook: `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-iq4r.md`.
Retired Swift1.5 w4dn8 era (payload purged 2026-10-17; rollback = HF re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`): `serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md`.
Retired fp8-native era (payload deleted 2026-10-15, rebuildable): `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-native-fp8.md`.
Retired FreeToken era: `serverAppConfigs-docs/LLM/frank-archon-freetoken-pr587.md`.
Retired vLLM era: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-archon-glm53-flash-w4a16.md`.
- Owns: `scripts/frank-archon.env`, `scripts/serve-frank-archon.sh`,
`scripts/frank-archon.service`, `scripts/ttm-preflight.sh`, template
`scripts/qwen38-froggeric-v22.5.jinja`; image digest-pinned
`stilldeadcode/radiance@sha256:4c3f4845…`; container name = unit name.
- Shape: TP=2 over `wht6`, `--max-num-seqs 3`, `--max-model-len 200000`,
`expert_tiered` placement, 44,928 MiB pinned host pool per rank, VRAM pair
`--vram-weights-mib 25600 --vram-kv-mib 5120` (ONE budget: weights + kv ≤
30,969 MiB/card), `fp8` KV, MTP depth 2, MNBT 2048, prefix cache 8 GiB host
+ 64 GiB disk, n-gram table pinned to RAM.
- Harness/gate contract: `models.yml` `contextWindow` MUST quote 200,000
(over-length is refused, never truncated); the ~147 GiB pinned-RAM ledger
is why this is a root unit, and `ttm-preflight.sh` raises
`ttm.pages_limit` itself because the kernel resets that knob to half of RAM
on every boot; `/health` binds only after the full load, so a bare health
gate is honest.
- Page: [[frank-archon]].
## Key Lessons Learned
Durable operational lessons from the fleet's history. Full detail in the
@@ -877,15 +557,13 @@ linked runbooks.
**ROCm / GPU**
- **Layer split was the deployed shape on both multi-device llama.cpp services,
for two different reasons.** On Dankserver it was the *only* way both R9700s
served one model (no collective could be dispatched on card #2) until the
2026-09-20 board swap; frank-medium now runs radiance TP=2 instead. On a
RCCL-less fork build it is also the fast one: tensor parallel falls back to
a butterfly allreduce and collapses prefill ~7x (measured 271.8 vs 1991 t/s).
Pin the spec-decode drafter to one device — an unpinned draft under tensor
mode aborts the KV build, and on a layer split the natural home is the card
that ends up holding the target's last layers.
- **Layer split needs no collective** — layers go in order and only the boundary
activation crosses PCIe, so it is the fallback wherever a collective cannot be
dispatched (and on an RCCL-less fork it beats tensor mode, which falls back to
a butterfly allreduce). Pin the spec-decode drafter to one device: an unpinned
draft under tensor mode aborts the KV build, and on a split its home is the
card holding the target's last layers. Deployed history:
[[frank-medium-llamacpp-dflash2]].
- **Bundling a ROCm runtime next to a llama.cpp binary: `cp -r SRC DST/` is a
trap.** With a missing `DST`, cp creates `DST` as a copy of `SRC` and drops
one directory level. rocBLAS and hipBLASLt look for their device code at a
@@ -1040,92 +718,44 @@ linked runbooks.
so acceptance drops on ALL traffic even where the matcher fires. Measure
chains per architecture, never by analogy.
- **A drafter's KV scales with the TARGET's context, and its window policy
decides how much that costs.** The draft context is built with
`cparams.n_ctx = llama_n_ctx(ctx_tgt)` (`common/speculative.cpp`) — there is
no flag to cap it — so a drafter that declares NO sliding window caches the
whole window on whatever device `--spec-draft-device` names. Swapping DFlash2
(`dflash.attention.sliding_window = 2048`, ~40 MiB of draft KV) for DSpark
(no window; 5 layers x 2 x 8 kv heads x 128 dim x 2 B = 20 KiB per position,
so **5.0 GiB at 262144**) put ROCm1 at 33.9 GB of 34.2 GB and killed the
FIRST draft graph capture with `ROCm error: out of memory` — after the model
had loaded and `/health` had answered. Read `sliding_window` out of the
drafter's GGUF header before sizing any spec-decode deployment, and prefer a
skewed `-ts` over a context cut when the full window matters. The pair was
swapped DFlash2 -> DSpark -> DFlash2 in one day (2026-09-16) and the round
trip moved the balance point FOUR layers (~4.3 GiB of headroom, 40:25 ->
36:29), which is the practical size of the effect. See
[[frank-medium-llamacpp-dflash2]].
decides the price.** The draft context is built at `llama_n_ctx(ctx_tgt)` with
no flag to cap it, so a drafter declaring NO sliding window caches the whole
window on `--spec-draft-device` and can OOM the first draft graph capture
AFTER `/health` has answered. Read `sliding_window` out of the drafter's GGUF
header before sizing, and prefer a skewed `-ts` over a context cut. Worked
example: [[frank-medium-llamacpp-dflash2]].
- **A draft context bills its own f16 KV over the WHOLE POOL, and on a
single-card endpoint that squeeze can cost more than the drafter wins.** The
draft context is built at the target's `n_ctx`, so with `--parallel N` the
drafter pays the POOL, not the slot: on frank-zealot's qwen35 pool
`--spec-type draft-mtp` adds **4,096 B/position of draft KV at the default
f16 dtype** (`--spec-draft-type-k/-v` can quantize the draft — untested here),
**3,072.00 MiB** at the 786,432 pool it shipped with and **2,048.00 MiB** at
the 524,288 pool it runs now, plus its own compute buffers (4,830 MiB all told
at the old pool). Booted arms, same card, same traffic (2026-10-18, every row
q8_0 KV): 3 slots + ubatch 2048 → **167.2 t/s decode with spec OFF, 109.2 with
spec ON**; dropping `UBATCH_SIZE` to 1024 hands the buffers back and spec ON
gives **213.5** (+28 % over the spec-OFF arm), and at 1–2 slots the SAME card
runs MTP at ubatch 2048 and still gains ~28 %. So the MTP question is never
"does the file carry the head" — it is "what do draft KV + buffers cost
against whatever the batch knobs keep for compute buffers", and that answer is
per-shape. Measure the arm, never the flag. Production corollary: beside
exact KV an **f16-draft** arm has no room — the trunk projects ~21,200 MiB of
the 24,123 addressable before the drafter is asked for anything; the untested
door is `--spec-draft-type-k/-v q8_0` (1,088 MiB). See
[[frank-zealot-llama-prism-bonsai9-mtp]].
- **A quant's scale overhead belongs in its byte rate, and doubling a measured
pool figure to "get the f16 number" invents 6.25 %.** This repo carried "f16
KV would bill 26,112 MiB" for weeks by doubling the q8_0 ledger figure
(13,056.00 MiB). q8_0 is 1.0625 B/element (32 codes plus one 2-byte scale per
32-block), f16 is exactly 2.0 ⇒ the true ratio is **1.882×**, so exact KV over
that pool is **24,576 MiB**. The conclusion (three f16 windows cannot boot a
card that leaves 24,123 MiB addressable) survived; the number did not — and
the honest rate is what made the resize calculable, because it shows 524,288
positions is the largest exact-KV pool that fits with room for the projector.
Derive byte rates from element count × block layout, then check them against
the ledger.
- **`--ubatch-size` has an OPTIMUM on the 2x R9700 layer split, not a plateau —
and the curve is BACKEND-SPECIFIC, so never carry it across (next bullet). Its
compute buffers land about TWICE as heavy on the card that holds the output
layers and the drafter.** Measured on frank-medium 2026-09-16 at the deployed
shape (262144 over two slots, a 64k prefill measured at 64k depth,
`scripts/bench-batch-ubatch.sh`): prefill runs 1046 tok/s at ubatch 256, 1068
at 384, **1193 at 512 (the peak)**, 1135 at 768, 1152 at 1024, 1151 at 1280,
1165 at 1536, 1148 at 1792 and 1078 at 2048. Both ends lose — below 512 the
SMs are not saturated, and 2048 is 10 % off the peak — and 512 -> 1536 costs
ROCm0 1.30 GiB and ROCm1 2.41 GiB to go 2.4 % SLOWER. The logical
`--batch-size` does not matter: 2048 through 8192 land within 2 % of each
other at a fixed ubatch and use identical VRAM. Decode never moves at all
(15.88 tok/s at depth 65536 on every cell). Because the buffers are
asymmetric, a split balanced at one batch is unbalanced at another: 512 ->
1536 shifted frank-medium's balance by a layer, so 36:29 became 35:30.
Re-measure the balance after changing the batch. (8192/2048 failed to allocate
its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1 had 0.08 GiB
free. That was a CONTIGUITY failure, not a total-bytes one, and no totals
model — the sweep's or the seam's — can see it.)
- **On the CUDA 3090 the same knob has NO interior optimum up to 2048, so
frank-zealot ships 4096/2048.** Measured 2026-10-05 by boot-pair A/B on the
live service: at 8,928 prompt tokens ubatch 1024 gives 3,972 / 4,077 t/s and
ubatch 2048 gives 4,207 / 4,212 — **+4.6 %** — for **616 MiB** (free 2,849 →
2,279). Decode did not move (91.9–93.0 vs 93.8–94.3, i.e. noise), and a
279-token prompt, which is below either ubatch, came back identical — that is
the mechanical proof the knob was the only variable, since a prompt shorter
than the ubatch cannot see it. The logical `--batch-size` was again free. And
the predecessor's "ubatch 2048 failed the scratch boot" was a shape
observation, not a law: the same file boots 2048 clean at 2 × 200,704. Bench
it by restarting the SERVICE through a transient `FRANK_ZEALOT_ENV_FILE`
drop-in, never a scratch port — one card holds one server. See
[[frank-zealot-llama-mimo-q6k-2x200704]].
single-card endpoint that squeeze can cost more than the drafter wins.** With
`--parallel N` the drafter pays the pool, not the slot, plus its own compute
buffers. So the MTP question is never "does the file carry the head" but "what
do draft KV + buffers cost against what the batch knobs leave for compute" —
a per-shape answer. Measure the arm, never the flag; the booted arm table is
on [[frank-zealot]].
- **A quant's scale overhead belongs in its byte rate; doubling a measured
figure to "get the f16 number" invents 6.25 %.** q8_0 is 1.0625 B/element
(32 codes + one 2-byte scale per 32-block), f16 is exactly 2.0 ⇒ the ratio is
**1.882×**, not 2×. Derive byte rates from element count × block layout, then
check them against the boot ledger — this repo carried an invented 2× number
for weeks. Current KV math: [[frank-zealot]].
- **`--ubatch-size` has an OPTIMUM, not a plateau, and the curve is
BACKEND-SPECIFIC — never carry it across hosts.** Its compute buffers land
about TWICE as heavy on the card that holds the output layers and the drafter,
so a split balanced at one batch is unbalanced at another: re-measure the
balance after any batch change. The logical `--batch-size` is nearly free.
A large-batch allocation can fail on BUFFER CONTIGUITY while totals still fit
— no totals model can see that. Ladders: [[frank-medium-llamacpp-dflash2]]
(R9700 split) and [[frank-zealot]] (CUDA 3090).
- **On the CUDA 3090 that same knob has no interior optimum up to 2048.** A
prompt shorter than the ubatch cannot see the knob at all — that is the
control that proves the knob, not the traffic, moved the number. When a bench
IS ordered, A/B it by restarting the SERVICE through a transient
`FRANK_ZEALOT_ENV_FILE` drop-in, never a scratch port: one card holds one
server. Numbers: [[frank-zealot]].
- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same
size.** qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is
worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on a full-attention one
(0.45 GiB of weights plus ~1.0 GiB of its own f16 KV at 262144) — both
measured on frank-medium. Balance by measurement, never by interpolating
between two splits; an affine fit reproduces the deployed point and overstates
the gaps in between. `tests/frank_medium_unit.bats` encodes that fit as a
gate. See [[frank-medium-llamacpp-dflash2]].
size** (a DeltaNet boundary and a full-attention boundary can differ by ~1 GiB
each with its own KV), so balance by measurement. An affine fit reproduces the
deployed point and overstates the gaps in between.
[[frank-medium-llamacpp-dflash2]].
- **ngram-mod prints no startup banner.** The load log shows the drafter's
`adding speculative implementation 'draft-dflash'` and nothing at all for the
matcher, so a chain can look unengaged while it is working. Identify it
@@ -1136,32 +766,17 @@ linked runbooks.
without acceptance.
- **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the
per-slot number is the contract — not `ctx-size`.** `n_ctx_slot =
CTX_SIZE / PARALLEL`, and llama.cpp refuses a prompt longer than the SLOT
(`400 exceed_context_size_error: request (184001 tokens) exceeds the
available context size (180224 tokens)`) even when the pool has room. So
frank-zealot's 2 → 3 reslice moved each conversation from the model's native
262,144 down to 180,224, and `models.yml` `contextWindow` had to follow or
the harness would discover the ceiling as a 400 on a long conversation. Side
effect worth knowing: the shorter slot made decode ~5 % FASTER (106.8 vs
100.4 tok/s) — less attention to scan and one fewer DeltaNet state resident.
The 2026-10-03 reslice went the other way (3 → 2 slots, 180,224 → 200,704,
pool 401,408) to fund unquantized f16 V, and `contextWindow` followed again;
decode landed 104.8 tok/s — the ~2 % off 106.8 is the exact-V tax. The same
day the freed space also bought the Q6_K rung (now 94.4 tok/s — the rung
itself is a further measured −10 %, weight-bandwidth, not slot geometry).
All moves measured; both slot directions cost or buy window, never pool.
See [[frank-zealot-llama-mimo-q6k-2x200704]].
CTX_SIZE / PARALLEL`, and llama.cpp refuses a prompt longer than the SLOT with
a 400 even when the pool has room, so `models.yml` `contextWindow` must follow
every reslice. Both slot directions cost or buy window, never pool; shorter
slots decode slightly faster (less attention to scan, one fewer recurrent
state). Measured reslices: [[frank-zealot]].
- **Price a VRAM budget from two boots, never from a formula — and an mmproj
costs ~1.5x its file size.** The frank-zealot 3090 fit was solved from two
measurements: 2 × 262,144 with no projector = 20,190 MiB used (KV 12,800 ⇒
**non-KV 7,390**), 3 × 196,608 with the f16 projector = 22,972 used (KV
14,113 ⇒ **non-KV 8,859**). The difference is the vision tower: a 918 MB file
that bills ~1.4 GiB, because the encoder's activation and compute buffers
ride with it. Subtracting the two rows is exact; extrapolating weights+KV+
"a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7. Then
solve `KV ≤ card − non-KV − headroom` and round to a multiple of
`PARALLEL × 4096`. Recorded in `scripts/frank-zealot.env` so the next resize
starts from measurement.
costs ~1.5× its file size** (the encoder's activation and compute buffers ride
with it). Subtracting two booted rows isolates it exactly; extrapolating
weights+KV+"a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7.
Then solve `KV ≤ card − non-KV − headroom` and round to a multiple of
`PARALLEL × 4096`. The two-boot table: [[frank-zealot]].
- **`images` and `image_url` are NOT interchangeable: only the content part
reaches the model.** `--mmproj` loads, `usage.prompt_tokens` rises, and the
model still invents an answer if the client used the legacy top-level
@@ -1326,65 +941,23 @@ linked runbooks.
## Runbook Index
Active:
One page per live endpoint; every payload swap it has ever run is a row in that
page's **Eras** table, with the rollback path in the same row. There is no
per-attempt page and no per-swap ledger anywhere else.
| Service | Runbook |
|---------|---------|
| frank-dragoon (flash-server 0.16.2, NATIVE v2 abliterated .hgn + MTP always-on, 2 slots over ONE 401,408 pool at ctx 200704, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md` |
| frank-zealot (llama.cpp PrismML fork build 10777, Bonsai-2-**9B** PQ2_0 **with the MTP head**, 3×262,144 q8_0 KV, MTP off by measurement, vision ON via Q8_0 mmproj, Dankclaw) | `serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai9-mtp.md` |
| frank-archon (radiance StillDeadcode, publisher Qwen3.8-Flash-Next iq4r .rad, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-iq4r.md` |
| Dankclaw kernel posture (`iommu=pt` since 2026-10-18, was `amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Service | Page |
|---------|------|
| `frank-archon` — radiance, 2× R9700 TP=2, Dankserver | `serverAppConfigs-docs/LLM/frank-archon.md` |
| `frank-dragoon` — halogen flash-server, Dankclaw iGPU | `serverAppConfigs-docs/LLM/frank-dragoon.md` |
| `frank-zealot` — llama.cpp (PrismML fork), RTX 3090 | `serverAppConfigs-docs/LLM/frank-zealot.md` |
| `frank-tts` — cloned-voice speech, Dankclaw | `serverAppConfigs-docs/Dankclaw/Frank TTS.md` |
| Dankclaw kernel posture (`iommu=pt` since 2026-10-18) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
Retired (rollback targets and history):
| Runtime | Host | Window | Runbook |
|---------|------|--------|---------|
| llama.cpp PrismML fork build 10754, Ternary-Bonsai-2-**27B** PQ2_0, 2×200,704 q8_0 K+V, MTP off (no head in the file), vision ON (`frank-zealot`) | Dankclaw | 2026-10-17 → 10-18 (rollback target — payload + 10754 fork tree on disk; owns the 34,816 B/token 16-full-layer KV law, the 25,088 MiB f16 OOM wall and the two-boot VRAM pricing) | `frank-zealot-llama-prism-bonsai2-pq20.md` |
| radiance Swift1.5-Qwen3.8-Flash-Next GPTQ w4dn8, 3×200,000 fp8-KV slab 25600 (`frank-archon`) | Dankserver | 2026-10-08 → 10-15 (rollback target — `.rad` purged 2026-10-17, byte-exact re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`; owns the slab-slope ladder, KV dtype law, KLD campaign, and every shape law the iq4r swap carried verbatim) | `frank-archon-radiance-swift15-w4dn8.md` |
| llama.cpp CUDA (master) Qwen3.5-9B-Uncensored Q6_K, 2×200,704 exact f16 KV MTP-off (`frank-zealot`) | Dankclaw | 2026-10-07 → 10-17 (rollback target — payload + master CUDA build on disk; owns the 32 KiB/token 8-full-layer KV law and the 2→3-slot reslice law) | `frank-zealot-llama-qwen35-9b-q6k-2x200704.md` |
| llama.cpp CUDA Gemma4-26B-A4B QAT Q4_K_M, 1×200,704 f16-K/q8_0-V MTP-off (`frank-zealot`) | Dankclaw | 2026-10-07 → 10-07 (same-day swap; rollback target — payload on disk; owns the `attention_k_eq_v` pair-billing KV law and the gemma4-assistant drafter-cost boots) | `frank-zealot-llama-gemma4-q4km-200704.md` |
| halogen-flash-server 0.16.2 abliterated Swift1.5 v2 `.hgn` + MTP always-on, TWO slots at ctx 262,144 over a 524,288 pool (`frank-dragoon`) | Dankclaw | 2026-10-06 → 10-17 (rollback target — payload + fetch script on disk; owns the 400-zone law and the sampler/prompt-cache contract) | `frank-dragoon-flash-server-v2-swift15.md` |
| ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-dragoon`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-dragoon-rocmfpx-q6agent-uncensored-mtp.md` |
| tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-archon`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-archon-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` |
| FreeToken PR #587 ROCm, Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 (`frank-archon`) | Dankserver | 2026-10-05 → 10-05 (rollback target — tag `frank-archon-freetoken-pr587-20261005`; trio + Dockerfile in the tag, image `localhost/frank-archon-freetoken:pr587-77a7201` and orcarouter payload on disk) | `frank-archon-freetoken-pr587.md` |
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K 2×200,704 f16 KV (`frank-zealot`) | Dankclaw | 2026-10-03 → 10-05 (rollback target — payload on disk; owns the ubatch-2048 A/B; the Ornith swap of 10-05 was committed but never deployed, so this was the LIVE payload Swift replaced) | `frank-zealot-llama-mimo-q6k-2x200704.md` |
| llama.cpp Ornith-1.5-9B Q8_0, ONE 262,144 f16 KV (`frank-zealot`) | Dankclaw | 2026-10-05 (committed 77decb5→380aafa, never deployed — superseded by Swift the same evening; payload on disk) | trail in `frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688.md` |
| llama.cpp CUDA Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2×114,688 draft-mtp q8_0 KV (`frank-zealot`) | Dankclaw | 2026-10-06 → 2026-10-07 (rollback target — payload on disk; owns the 3090 ubatch ladder A/B and the 34 KiB/token qwen35 KV law) | `frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688.md` |
| ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-dragoon`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-dragoon-rocmfpx-q8agent-mtp.md` |
| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-dragoon`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-dragoon-gufo-qwen38-27b.md` |
| vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-dragoon`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-dragoon-vllm-mimo-v26-bf16.md` |
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-zealot`) | Dankclaw | 2026-10-01 → 10-02 (the rung returned ACTIVE 2026-10-03 at the 2×200,704 shape — `frank-zealot-llama-mimo-q6k-2x200704.md`) | `frank-zealot-llama-mimo-q6k.md` |
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q5_K_M 2×200,704 f16 KV (`frank-zealot`) | Dankclaw | 2026-10-02 → 10-03 (rollback target — one `MODEL_FILE` line, buys back 875 MiB + the −10 % rung tax; owns the boot-pair table) | `frank-zealot-llama-mimo-q5km.md` |
| SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-zealot`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-zealot-sglang-mimo-v26.md` |
|tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-archon`)|Dankserver|2026-09-30 → 10-01|`frank-archon-glm53-flash-w4a16.md`|
| llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-dragoon`) | Dankclaw | 2026-09-21 → 09-28 | `frank-dragoon-unsloth-qwen38-27b-q8xl.md` |
| R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-archon`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-archon-r9v-radiance-qwen38-uncensored.md` |
| llama.cpp b11030 Qwen3.8-Flash-Next Q8_0 + shared MTP (`frank-archon`) | Dankserver | 2026-09-26 → 09-27 | `frank-archon-llamacpp-qwen38-flash-next-q8.md` |
| ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-dragoon`) | Dankclaw | 2026-09-15 → 09-16 | `frank-dragoon-ik-uncensored.md` |
| vllm-radiance TP=2 MXFP4 + DFlash2 (`frank-medium`) | Dankserver | 2026-09-20 → 09-26 | `frank-medium-radiance-mxfp4.md` |
| Lucebox DS4.1 DeepSeek-V4.1-Flash ROCmFP23 (`frank-archon`) | Dankserver | 2026-09-26 (same-day swap) | `frank-archon-lucebox-ds4.md` |
| kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` |
| halogen-flash-server Qwen3.8-Flash-Next w4b (`frank-dragoon`) | Dankclaw | 2026-09-16 → 09-19 | `halogen-flash-server.md` |
| CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-dragoon`) | Dankclaw | 2026-09-19 → 09-21 | `frank-dragoon-ciru-orca.md` |
| unsloth llama.cpp b11030 Qwen3.8-Flash-Next UD-IQ4_XS (`frank-dragoon`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-dragoon-unsloth-qwen38-flash-next.md` |
| Kairic Edge Qwen3.8-27B IU4 W4A4 (`frank-dragoon`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-dragoon-kairic-edge-qwen38-27b.md` |
| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-zealot`) | Dankclaw | 2026-09-16 → 09-17 | `frank-zealot-qwen38-gsq-rco.md` |
| vLLM Gemma4-12B w4a16 (`frank-zealot`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-zealot-vllm-gemma4.md` |
| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-zealot`) | Dankclaw | 2026-09-17 → 10-01 | `frank-zealot-gemma4-12b.md` |
| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
| eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` |
| Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` |
| CIRU ActiveFPX v3 / PromptForge | Dankclaw | through 2026-08-22 | `ciru-activefpx-promptforge.md` |
| DFlash2 Vulkan | Dankclaw | 2026-08-21 (same-day revert) | `dflash2-large-vulkan.md` |
| FastFlowLM NPU (`frank-npu`) | Dankclaw | twice, retired 2026-09-11 | `frank-npu.md`, `FastFlowLM NPU Inference.md` |
| FreeToken (official + Huihui FP8, `frank-zealot`) | Dankserver | 2026-09-02 → 09-15 | `freetoken-frank-small.md` |
| FastMTP patched llama.cpp | Dankserver | retired 2026-08-18 | `qwen3.8-spec-matrix.md` |
| ik_llama.cpp Ornith CRACK | Dankserver | retired 2026-09-02 | `ikllama-ornith-dankserver.md` |
| llama.cpp layer split + DFlash2 (`frank-medium`) | Dankserver | 2026-09-16 → 09-20 | `frank-medium-llamacpp-dflash2.md` |
Retired runbooks are under `serverAppConfigs-docs/LLM/`. Retired scripts live
in `old/` and `.AGENTTRASH/`.
Retired **endpoints** (not payloads) keep their own pages — `frank-medium`,
`frank-npu`, the pre-name Dankclaw/Dankserver eras — because no live page owns
them; `serverAppConfigs-docs/index.md` is the locator for every page in the
vault. Retired scripts live in `old/` and `.AGENTTRASH/`.
## Why
@@ -1402,7 +975,9 @@ history.
| Unstaged local changes | **Commit or discard** — check `git status` before starting work; ask if unsure. |
| Deploying a committed change | **`scripts/deploy.sh <target>`** — pushes, pulls ff-only, plans, dispatches, health-checks. |
| Container without a Quadlet unit | **Give it one** — boot-persistent, self-healing `.container` via git. |
| Benchmarking | **Stop and mask the service first**, bench, unmask, restart, verify health. |
| Changing a model/payload | **Swap it** — edit the trio, deploy, `health` + one chat/tool/vision probe. No benches unless Frank orders one; if it misbehaves, revert. |
| Writing a doc | **One owner per fact** — edit the endpoint/service page; everywhere else links. Never restate a number. |
| Benchmarking (only when ordered) | **Stop and mask the service first**, bench, unmask, restart, verify health. |
| Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
| Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. |