Compare commits

...
Author SHA1 Message Date
Frank 2def6282b5 docs(frank-archon): rollback re-key after 2026-10-17 disk purge
The Swift1.5 w4dn8 .rad, the GLM-4.7 GGUF, and every derived model cache
were purged from dankserver (trash-staged, emptied with approval; 763 GiB
free). Rollback of the iq4r swap is now an HF re-download of the
byte-exact frank8465 mirror (LFS sha256 09dd9306...). Corrects the w4dn8
runbook header (still said LIVE) and rebuilds both Rollback sections;
log.md carries the purge record.
2026-10-17 21:00:26 -07:00
Frank 986bbc9e36 docs(frank-zealot): cutover deployed — production ledger, live probes, picker row
- production boot reproduced the census (22,664/1,461, fit 21,439 leaving
  2,119 >= 1,872, KV 13,328.00 MiB at n_ctx_seq 200,704, n_slots 2), live
  probes green (pong + reasoning_content, tool_calls, 512px -> Red, 75.8 t/s),
  e2e 33/33
- records the two live-path stragglers the swap had to catch: the serve
  script's retired-payload DEFAULTS and the OpenWebUI model row description
  (live DB state, re-keyed with a backup, no forced restart)
2026-10-17 19:48:48 -07:00
Frank f48b45c1fa frank-zealot: sweep the live path — serve defaults, unit text, picker docs
- serve script: header + every fallback default re-keyed to the Bonsai-2 trio
  (vendor LLAMA_BIN, PQ2_0 MODEL_FILE, Q8_0 MMPROJ) and CACHE_TYPE_*:=q8_0 —
  a defaulted run must land on the shipped shape, never on the retired 9B/f16
  pair (the trial window proved a silent default is how a swap goes wrong)
- unit Description now names the payload + KV dtype that `systemctl status`
  will show during the next incident
- OpenWebUI decoration-row description re-keyed, with the law that the row is
  live DB state a swap must follow into the database
2026-10-17 19:42:13 -07:00
Frank 322ebb1658 docs(AGENTS): fleet topology + llama.cpp laws for the PQ2_0 zealot era
- topology row + zealot section: PrismML vendor build, Bonsai-2 PQ2_0,
  q8_0 KV, measured ledger; runbook index active/retired rows
- new durable laws: custom-tensor-type runtime door (vendor build is forced),
  mmproj outside the engine fit projection, first-request warmup artifact,
  thinking-budget probe traps + load-bearing reasoning_effort fold
2026-10-17 19:41:11 -07:00
Frank 7706e06775 docs(frank-zealot): Prism Bonsai-2 PQ2_0 runbook; indexes and log re-keyed
- new runbook: runtime lock (build 10754 / type 142 door), geometry census,
  four-arm VRAM table (f16 wall, q8_0 deployed, q4_0 rung, 2x262,144),
  measured rate ladder, behavior contract, ops notes
- LLM/index.md + vault root index: new ACTIVE entry, 9B era retired in place
- log.md: the trial+cutover wave record, incl. the env-var-name law
2026-10-17 19:41:11 -07:00
Frank f383038437 homepage: zealot card -> Ternary-Bonsai-2 PQ2_0 (censored base)
- card/docs: prism-ml payload link, PQ2_0 spec, ~77 t/s (measured single
  stream, replacing the 9B's ~95)
- warning re-keyed: zealot is no longer an uncensored endpoint, so the card
  carries the base-alignment notice and points uncensored work at dragoon;
  the test's "common uncensored text" equality (zealot==dragoon) is retired
  with the payload it described — dragoon alone carries the legal warning
2026-10-17 19:40:56 -07:00
Frank 64185d63b9 frank-zealot: cutover to Ternary-Bonsai-2 PQ2_0 with q8_0 KV
- env: Bonsai2 payload pair, CACHE_TYPE_K/V=q8_0 (f16 OOMs at 2x200,704:
  34 KiB vs 64 KiB/token over the 16 full-attention layers), measured boot
  ledger 22,664 used / 1,461 free incl. the 873 MiB mmproj that lands AFTER
  the engine fit check, q4_0 fallback rung, CRACK one-line MODEL_FILE swap
- deployer: runtime/model/mmproj gates re-keyed to the PrismML vendor build
  (the only loader for GGML tensor type 142 + prism.hadamard.* metadata)
- seam: payload trio pinned; KV argv assertions now derived from CACHE_TYPE_*
  in the env file — the wrong-var-name f16 fallback that produced the first
  failed trial window can no longer pass silently
2026-10-17 19:40:56 -07:00
Frank 430c42ab18 frank-zealot: Bonsai2 PQ2_0 candidate fetch script (official LFS shas; CRACK variant documented, unpinned) 2026-10-17 19:40:56 -07:00
dank 3ccc1e554f Update README.md 2026-10-15 18:22:22 -07:00
dank 7a69c31128 Update README.md 2026-10-15 18:14:53 -07:00
Frank f96898d4c5 docs(authentik): pk 22-23 provisioned — mitch + eric in the human-user template 2026-10-15 15:20:15 -07:00
Frank 3e140cd077 docs(log): iq4r swap cutover record — e2e 33/33 after gluetun boot-race repair 2026-10-15 05:53:42 -07:00
Frank c5ac0725f7 docs(frank-archon): iq4r first-boot ledger — pin 1.0.13 loads w4nl, geometry verbatim
kv_attn 26.00 KiB fp8 pages at 607,740 tokens (identical to the w4dn8
boot), slab 17,610/17,658 slots per rank (above w4dn8's 17,548 — ten
bf16-protected experts per layer pack more of the 4-bit table), sampler
9 rows, MTP 2/step, KV lending 4.22 GiB, prefix tier recovered 900,176
blocks across the payload swap. Vision probe green (reasoning read the
solid-red PNG; empty content at 64 tokens = the thinking-budget law).
e2e gate RED on three dankclaw specs: Marreta was down since the 01:41
dankclaw reboot (marreta-browser dependency-failed at boot beside
gluetun, not re-armed); pair restarted manually, redeploy rerunning
the gate.
2026-10-15 05:44:09 -07:00
Frank f44c39812f docs(frank-archon): iq4r era runbook + index/AGENTS re-key
New runbook frank-archon-radiance-qwen38-iq4r.md (publisher container,
payload chain, shape-verbatim cutover record with the boot-ledger and
measurement slots left open for the first boot, rollback). LLM index,
root vault index, and AGENTS.md (fleet table row, section, runbook
index: iq4r ACTIVE, w4dn8 RETIRED/on-disk, fp8-native deleted). Full
suite green.
2026-10-15 04:16:28 -07:00
Frank 6dcf4a5913 llm(frank-archon): swap payload to the publisher's iq4r .rad, shape verbatim
Model swap on the radiance endpoint: MODEL_FILE moves from the in-house
qwen4exp-swift15-w4dn8-i8-hc8m.rad (116,416 MiB) to the engine author's
published qwen3.8-next-flash-fp8-iq4r-moe.rad (116,322 MiB, official
Qwen3.8-Flash-Next bf16 -> 4-bit non-uniform codebook experts + int8
trunk, card KL 0.0789 / top-1 92.85 % vs bf16). NO argument changed:
25600/5120 vram pair, 44928 host pool, fp8 KV, 3 x 200000, MTP depth 2,
wht6, MNBT 2048, froggeric v22.5 — the 2026-10-13 shape carries verbatim.

New fetch script scripts/download-frank-archon-iq4r.sh (hfget + exact
byte count 121,969,901,568 + LFS sha256, --verify-only default, ungated,
no token on the command line). Unit/serve/deployer lore re-keyed; the
seam pins the new payload id + sha256 and keeps the w4dn8 rollback
provenance. Homepage card points at the publisher repo.

Rollback: revert this commit (w4dn8 .rad stays on disk). The fp8-native
rad was deleted from dankserver 2026-10-15 (Frank's one-time rm, volume
for this payload); rebuild path documented.
2026-10-15 04:10:50 -07:00
Frank a9d45e7012 frank-archon: the slab is a threshold, and decode is where slots cost
Measured, 512 tokens, greedy, cold, at the deployed 3 x 200,000 fp8 shape:
prefill 22,208 -> 3,397, 24,000 -> 3,290, 25,600 -> 3,290 t/s at 43.5k —
the third slab step bought nothing, so the slab is a threshold whose cliff is
the 2026-10-11 arm at 26,071 (4,379), not a slope. Decode ladder: 99.0 t/s
single-stream at 43.5k, 93.3 at 197k (nearly window-free), and 3 concurrent
streams 113.9/43.3/10.9 = 18.4 t/s aggregate — the slot count is paid in
decode concurrency, the axis the shape hunt never measured because probe leg
B generates 24 tokens. probe-frank-archon-kv.py gains leg D (N-stream decode,
512 tokens, per-stream + aggregate) so no resize is priced on prefill alone.
2026-10-13 20:51:32 -07:00
Frank 2581b8d4b6 docs(frank-archon): the slab is a slope, and the ceiling was mine not the engine's
Measured A/B on the shipped pair 25600/5120: A-cold 43.5k prefill 3,290 ->
3,584 t/s (+8.9 %) and the 3-way 197k aggregate 3,051 -> 3,589 t/s (+17.6 %,
wall 193.7 -> 164.5 s). The prediction I shipped with the arm -- slab is a
threshold below ~26 GiB, so 1,600 MiB buys ~0 -- is falsified by the measurement
it asked for. Two slab points inside noise of EACH OTHER (22,208 / 24,000) are
not a plateau. The cliff is above 25,600: 26,071 measured 4,379 t/s.

Ceiling re-derived from the engine's own budget table (card 32,624 MiB, pair at
29,120 settling 2,058 MiB free against its 209.19 MiB floor => 30,969 claimable).
The 29,120 bound was my 1 GiB \`keep free\` reserve; the seam now derives the
number, so the next resize starts from the engine's arithmetic.

Ledger for the record: slab 16,235 -> 17,548 slots while the pinned host pool
hands the same experts back (8,893 -> 7,580) -- the transfer is the mechanism,
the pinned pool is the slow path. Capacity contract live-verified at 3 slots:
3 x 199,984-token prompts accepted (599,952 tokens), fourth refused as too
large to ever fit.

Probe law learned: the oversized-prompt refusal names the prompt's token count
BEFORE prefill (free calibration) and fires on prompt tokens ALONE, not on
prompt + max_tokens; the window is ~514,286 chars at 2.5714 chars/token.
2026-10-13 20:33:02 -07:00
Frank 09a0251d2d llm(frank-archon): claim the card's settling slack, slab 24000 -> 25600
The 2 GiB/card that \`rocm-smi\` shows free is not idle: the engine's budget
table calls it driver context + the 896 MiB activation arena + 96 MiB headroom,
and says the rest is `throughput on the floor`. Its own settle line is the
number that matters — `card 0 settled at 2.01 GiB free against 209.19 MiB (the
96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the static region)` — so
1,849 MiB above the ENGINE's floor is claimable and the real ceiling is 30,969
MiB, not the 29,120 my 1 GiB reserve invented on 2026-10-11.

Shipped pair 25600/5120 = 30,720: +1,600 MiB of slab, ~458 MiB above the floor.
The slab is already saturated at 24,000 (mover took all of it, 451 KiB slack),
so this is the only direction left — KV is at its 3-slot floor (5,063 needed of
5,120).

Named prediction, because today's two points say the slab is a THRESHOLD and not
a slope (22,208 -> 3,397 t/s, 24,000 -> 3,290 t/s at 43.5k single-stream): this
arm buys ~1,600 resident expert slots and ~0 prefill. The point that moves
prefill is 26,071 (2026-10-11: 4,379 t/s), fundable only at two slots. Prefill
moving > +-3 % here falsifies the threshold reading.

Seam: the ceiling gate now derives from the engine's budget table (30,969) and
is the single owner of the pair arithmetic; the duplicate I put in the shape
test is removed.
2026-10-13 20:18:41 -07:00
Frank 066813ea95 llm(frank-archon): 3 x 200,000, pool cut to the contract, slab +1,792 MiB
Frank's call: three slots, and spend the KV the third slot doesn't need on
resident experts. `--max-num-seqs 3`, `--vram-kv-mib 6912 -> 5120`,
`--vram-weights-mib 22208 -> 24000` — the pair is still exactly the measured
29,120 MiB/card ceiling. The pool funds 608,027 tokens against the 600,000-token
contract (3 x 200,000, +1.3 %), and the slab gains 8.1 %.

The direction is the 2026-10-11 A/B: prefill is slab-limited, KV beyond the
contract buys nothing. At fp8 the slab curve measured on this host is
22,208 -> 3,397 t/s and 26,071 -> 4,379 t/s at 43.5k, i.e. ~0.25 t/s per MiB, so
24,000 predicts ~3,850 t/s single-stream. Falsifiable; the probe is the check.

The cost is named, not hidden: a FOURTH full-height conversation no longer fits
(4 x 200,000 needs 6,601 MiB of pool) and is refused as too large to ever fit —
radiance never truncates. Reverting to four slots is one env pair.

Seam: two new arithmetic gates, because both failures are boot-then-400 shape
faults that no flag check catches — the pair must respect the per-card ceiling,
and the pool must back seqs x max-model-len at the dtype's measured rate plus
the dtype-independent terms (110 MiB a sequence of recurrent state, 448 MiB of
prefix checkpoints).
2026-10-13 20:00:39 -07:00
Frank 7daa886dd6 docs(frank-archon): work-log entry and the 4-way capacity probe
Probe on the live service: four concurrent 196,960-token chats, 4/4 exact
marker retrieval, 787,842 tokens in 238.4 s (3,305 t/s aggregate prefill = the
single-stream ceiling, so the four streams shared one pipeline and the KV pool
never ran short); prefix-cache repeat answered in 1.0 s with cache_n=43008.
Recorded with the caveat that 8 generated tokens do not make a decode rate.
2026-10-13 19:30:46 -07:00
Frank bae78e7dae docs(frank-archon): the ttm reset outage, and the 4 x 200,000 fp8 shape
Runbook gains the 2026-10-13 section (symptom signature, the runtime-knob law,
the shape arithmetic, the boot ledger that confirms it: 843,276 tokens =
4 x 200,000, 12 all-reduce rows, 5.87 GiB of KV lent to the slab). AGENTS.md
gains the fleet-row update, the KV dtype law at four slots, the sysfs-raise
lesson under Key Lessons, and the pinned-RAM paragraph now points at
scripts/ttm-preflight.sh as the repair. LLM index entry + topology node match.
2026-10-13 19:28:25 -07:00
Frank a6f55f4cc6 fix(frank-archon): raise the ttm knob from a root ExecStartPre — User=dank cannot write it
a9da65e put the raise inside the serve script, which runs as `User=dank`: the
knob is root-owned 0644, so `-w` is false and the write silently failed — the
gate printed its refusal on a cap it had just failed to move. The privileged
path is systemd's `+` prefix: `ExecStartPre=+scripts/ttm-preflight.sh --raise`
runs as root even under `User=`, before ExecStart, so `Restart=always` can
never outrun it. The lib is now also executable (`--raise` sources the env file
and verifies), and keeps a `sudo -n` fallback for a human running the gate from
a shell.

Seam: the refusal cases run with a PATH that has `cat`+`bash` and NO `sudo`
(root writes a 444 file, so a passwordless dev box would have tested nothing),
and both now assert the gate's own exit 1, not merely non-zero. New seam covers
the unit line and the `--raise` entry point end to end (writable knob -> raised,
no args -> usage exit 2).
2026-10-13 19:20:33 -07:00
Frank a9da6589a6 fix(frank-archon): :" guard needs a space — :"${VAR:?}"` runs :204800
The TTM_TARGET_MIB guard I added on 1a6a78e was written `:"...` (no space), so
bash parsed the expansion as one word and executed `cmd :204800` as the
command `:204800`: exit 127 `command not found` on every start, before the
ttm gate ran. Caught on the deploy: the unit crash-looped 34 times in ~4 min
with exactly that journal line.

probe-frank-archon-kv.py now reads the slot count, window and KV dtype from the
deployed env on the host, so probe B exercises the capacity contract that is
actually live (4 x 200,000 today) instead of a hardcoded pair, and each probe's
marker starts with a unique word because show() scores the hit on that word.
2026-10-13 19:16:48 -07:00
Frank 1a6a78ef73 fix(frank-archon): raise the ttm pages cap in the serve path; reshape to 4 x 200,000 fp8 KV
The 2026-10-12 23:17 power failure reset /sys/module/ttm/parameters/pages_limit
from the manual 2026-10-05 raise (200 GiB) back to the kernel default (half of
RAM = 125,988 MiB), below the pinned-RAM ledger floor of 150,000 MiB. The serve
script's preflight named the refusal and exited 1; Restart=always re-ran it
6,919 times over ~20 hours and frank-archon served nobody.

scripts/ttm-preflight.sh (sourced by the serve script) now RAISES the knob
itself before refusing: the unit is root, the knob is a 0644 root module
parameter, and the cap is a runtime knob whose default is half of RAM — so a
manual sysfs write is not a repair procedure, it is a countdown to the next
reboot. FRANK_ARCHON_TTM_TARGET_MIB=204800 records the host ceiling; the gate
re-reads the cap after writing and only then refuses, so a knob the kernel
will not move is still a named failure.

Shape change (Frank's call): FOUR decode slots x 200,000 tokens with fp8
(quantized) KV. The pool must back seqs x max-model-len, so fp8 is what funds
the fourth slot: at bf16 the four slots cost 11,679 MiB of pool and would cut
the resident expert slab to 17,441 MiB (-33 %); at fp8 they cost 6,601 MiB, so
the pool stays 6,912 MiB (predicts 843,566 tokens), the VRAM pair stays at the
measured 29,120 MiB/card ceiling, the weights slab and the 44 GiB host pool are
untouched, and the fused TP all-reduce rides 4 x (1+2) = 12 rows, inside the
25-row bound where the wht6 wire is bit-exact.
2026-10-13 19:09:58 -07:00
26 changed files with 2007 additions and 461 deletions

No files matched your search

+216 -113
View File
@@ -25,9 +25,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | ukisai/Swift1.5-Qwen3.8-Flash-Next BF16 (335.3 GiB, 397/397 sha256-verified) converted by `scripts/convert-swift15-w4dn8.sh` to `qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB**): **all 49,152 experts GPTQ w4** (10M-token calibration on engine 1.2.0) + **24 whole bf16-protected experts/layer** (energy-share law), int8 trunk/lm_head, fp8 HC, PLE n-gram fp8 fixed-scale, on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, explicit `--vram-weights-mib 26071 --vram-kv-mib 3543`), **47.7 GiB n-gram copy pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV** (inherited; pool 3.46 GiB), **MTP spec decode depth 2**, `--max-num-seqs 3`, `--max-model-len 200000`, MNBT 2048, Froggeric v22.5 template. KLD-gated (mean KL 0.0131, top-1 97.92 % vs BF16); decode ~190 t/s live (gate bar 30.8). Cutover 2026-10-08 from fp8-native (payload on disk = rollback). | 7770 |
| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | **Publisher payload** (swap 2026-10-15, MODEL_FILE-only): `StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe` → **`qwen3.8-next-flash-fp8-iq4r-moe.rad`** (121,969,901,568 B = **116,322 MiB**, Hub-LFS sha256 `0af5e962…` verified at fetch by `scripts/download-frank-archon-iq4r.sh`) — the OFFICIAL Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's author: routed experts as **4-bit codes into a 16-level non-uniform codebook** (128-pt Walsh–Hadamard rotation, GPTQ on a 10M-token calibration + 3 coordinate-descent sweeps, E4M3 scales per 64 weights), **10 whole bf16-protected experts/layer**, int8 W8A8 trunk/lm_head, fp8 HC mixtures, fp8 fixed-scale PLE n-gram (47.7 GiB), **MTP head + bf16 vision tower in the container**; card KL 0.0789 mean all-tokens / top-1 92.85 % vs BF16. **Base-alignment (censored) model** — the abliterated family lives on frank-dragoon. Shape carried VERBATIM from the 2026-10-13 record: **3 × 200,000 fp8-KV** (`--max-num-seqs 3`), **25,600 MiB slab + 5,120 MiB pool = 30,720 of the 30,969 MiB claimable**, 44 GiB/rank pinned host pool, PLE pinned to RAM, MTP depth 2, MNBT 2048, Froggeric v22.5 template. Decode/prefill ladders in the runbook are the **w4dn8 record** (the bar to beat; new-payload probes pending). Rollback: re-download the w4dn8 `.rad` from the HF mirror `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` (LFS sha256 `09dd9306…`) + revert the swap commit — the local copy was **purged 2026-10-17** (it was the last non-iq4r model on the host; the GLM-4.7 GGUF, the radiance kvcache tiers, and every stale compile cache went in the same purge, 763 GiB free). The fp8-native `.rad` + the raw Swift1.5 BF16 source were DELETED from disk 2026-10-15 (Frank's one-time rm, volume for this payload; rebuildable via the fetch/build scripts). | 7770 |
| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 262,144 ctx** (KV pool 524,288 positions ≈ 14.0 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), harness window 217,088 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold)|7770|
| Dankclaw | `frank-zealot` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** (7,359,259,008 B; dense `qwen35` hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP tensors in the file**) + the publisher's own **BF16 mmproj** (921,704,448 B) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **exact f16 K + f16 V** under flash attention (32 KiB/token over the 8 full layers = 12.25 GiB pool), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (forced: no `nextn` tensors exist). Priced from the MiMo Q6_K boots (same geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048. Cutover 2026-10-07 (late) from Gemma4-26B-A4B Q4_K_M 1×200,704. All three endpoints route harness **compaction** here. | 7771 |
| Dankclaw | `frank-zealot` | **llama.cpp PrismML fork build** (vendor release `0.2.0-dev` build **10754**, commit `2459f68b5`; root systemd unit, relinked — the ONLY loader for this payload) | `prism-ml/Ternary-Bonsai-2-27B-gguf` **PQ2_0** (7,206,168,928 B, sha256 `3907dc16…`; `qwen35` hybrid: **64 blocks, 16 full-attention + 48 Gated DeltaNet**, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP head in the file**; `general.file_type 141` = custom tensor type **142** + `prism.hadamard.*` dequant metadata, which is what locks the runtime) + the publisher's own **Q8_0 mmproj** (629,246,976 B; Qwen3-VL tower, 27 blocks, 768 px, `qwen3vl_merger`) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **q8_0 K + q8_0 V** under flash attention (34 KiB/token over the 16 full layers = 13,328 MiB pool; **f16 KV needs 25,088 MiB and cannot boot this card**), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (no head in the file). Boot 22,664 used / 1,461 free; 76.6 t/s single-stream decode, 691.7 t/s prefill at 166k. Cutover 2026-10-17 from Qwen3.5-9B-Uncensored Q6_K (same shape; payload + master CUDA build on disk as rollback). |
Retired endpoints (`frank-npu`, and every prior runtime) are in
the [runbook index](#runbook-index). The full swap history lives in
@@ -429,117 +429,163 @@ Rollback: revert the swap commit and redeploy — the 27B payload
halogen 0.1.4 image digest all remain on disk; the ROCmFPX door behind
that (`scripts/build-rocmfpx.sh`, pin `36e9acd`) is untouched.
### frank-zealot — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
### frank-zealot — llama.cpp (PrismML fork) on the RTX 3090 (Dankclaw, ACTIVE)
**llama.cpp master** (0.5.0-dev, commit `5fc4f3c`; CUDA sm_86 build
`/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`,
`GGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit**
**The PrismML fork build of llama.cpp** (vendor release, `--version` =
`0.2.0-dev (build 10754, commit 2459f68b5)`;
`/home/dank/GitPackages/llama-prism/llama-server`) as a **root systemd unit**
(`scripts/frank-zealot.service`, `relink_unit`ed into
`/etc/systemd/system/`), serving
`HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive` at **Q6_K**
(7,359,259,008 B, sha256 `c0ba7beb…` computed locally — the Hub carries no
LFS oid) **plus the publisher's own BF16 mmproj** (921,704,448 B), both
`/home/dank/models/Qwen35-9B-HauhauCS-Q6K/`, on port 7771. Cutover
2026-10-07 (late) from the Gemma4-26B-A4B Q4_K_M 1×200,704 shape (same day).
`/etc/systemd/system/`), serving `prism-ml/Ternary-Bonsai-2-27B-gguf` at
**PQ2_0** (7,206,168,928 B, sha256 `3907dc16…` = the Hub LFS oid) **plus the
publisher's own Q8_0 mmproj** (629,246,976 B), both
`/home/dank/models/Bonsai2-PQ2_0/`, on port 7771. Cutover 2026-10-17 from the
Qwen3.5-9B-Uncensored Q6_K era (same shape).
**Architecture: qwen35 dense hybrid** — read from the GGUF's own header, not
the model card: 32 layers, `full_attention_interval = 4` ⇒ **8 full-attention
layers (3, 7, 11, …, 31) + 24 Gated DeltaNet linear layers**, 16 attention
heads / **4 KV heads** × 256 dim, `ssm` conv_kernel 4 / state_size 128 /
group_count 16 / time_step_rank 32 / inner_size 4096, native 262,144, vocab
248,320. Tensor types are Q6_K × 250 + F32 × 177 — standard quants only, so
the stock master build loads it (arch `qwen35` is registered in this build's
`src/llama-arch.cpp`). **KV accounting:** the pool scales with the 8 full
layers only — 8 × 4 × 256 = 8,192 elements/token per cache, so f16 K + f16 V
bills **32 KiB/token** = 12.25 GiB at the 401,408 pool; the 24 DeltaNet
layers bill a fixed recurrent state per SLOT, not per token, which is exactly
why two 200k slots fit where the 26B MoE fit one.
**The runtime is part of the payload identity.** The file is
`general.file_type 141` (PQ2_0, 2.13 bpw, group 128) with **402 tensors of the
custom GGML type 142** plus `prism.hadamard.*` metadata (normalized
Sylvester–Walsh transform, block 1024, 401 named weights, inverse on
`token_embd`, `gdn_v_grouped`). Stock llama.cpp refuses type 142 at load, and a
loader that ignored the Hadamard metadata would dequant wrongly — same lock class
as the ROCmFPX era. Retargeting the payload without the fork ships an endpoint
that cannot load, so the seam pins both paths.
**The shape (2026-10-07):** `--ctx-size 401408 --parallel 2` = TWO
**Architecture: qwen35 hybrid** — read from the GGUF's own header: **64
blocks**, `full_attention_interval = 4` ⇒ **16 full-attention + 48 Gated
DeltaNet layers**, 24 attention heads / **4 KV heads** × 256 dim, embedding
5,120, ffn 17,408, `ssm` conv_kernel 4 / state_size 128 / group_count 16 /
time_step_rank 48 / inner_size 6,144, native 262,144, vocab 248,320. **KV
accounting:** the pool scales with the 16 full layers — 16 × 4 × 256 = 16,384
elements/token per cache, so **q8_0 K + q8_0 V bills 34 KiB/token** = 13,328
MiB at the 401,408 pool (ledger-confirmed to the MiB), while **f16 KV needs
25,088 MiB and cannot boot on a 24,123 MiB card** (measured `cudaMalloc` OOM).
The 48 DeltaNet layers bill a fixed recurrent state per SLOT — ~150 MiB each —
so pool arithmetic alone under-prices the shape.
**The shape (2026-10-17):** `--ctx-size 401408 --parallel 2` = TWO
conversations of 200,704 (Frank's cap; the harness `contextWindow` quotes
the per-slot number), `--flash-attn on`, **f16 K + f16 V** (exact KV),
batch 4096 / **ubatch 2048** (the measured CUDA optimum: +4.6 % prefill,
decode flat, 616 MiB — MiMo-era A/B on this card), `--log-verbosity 4`,
`--mmproj` for vision, **speculative decoding OFF** (`SPEC_TYPE=none`).
the per-slot number), `--flash-attn on`, **q8_0 K + q8_0 V**, batch 4096 /
**ubatch 2048** (the measured CUDA optimum carried from the MiMo/Swift A/Bs on
this card), `--log-verbosity 4`, `--mmproj` for vision, **speculative decoding
OFF** (`SPEC_TYPE=none`). Boot: **22,664 MiB used / 1,461 free**; the
projection (21,439) excludes the projector's 873.10 MiB worst-case, which
lands AFTER the fit check — read settled `nvidia-smi`, never the projection.
Measured: 76.6 t/s single-stream decode (the 9B ran 94.4), 1,258 t/s prefill
at 35k, 691.7 t/s at 166k, 109.4 t/s aggregate across two concurrent streams.
The **first request after boot is a warmup artifact** (2.6 t/s prefill / 37.7
t/s decode measured) — never report it.
**Why MTP is off**: it is not a choice — **this Q6_K file carries no
`nextn`/`mtp` tensors** (verified from the tensor-name list), and MTP is a
property of the file, not of the flags. The publisher card advertises "MTP
support" for the base architecture; the quantized export dropped the head.
**Why MTP is off**: the PQ2_0 export carries no MTP head (upstream ships a
separate `…-CRACK-MTP-GGUF`, +451 MB, for that), and MTP is a property of the
file, not of the flags.
**Measured boots** (nvidia-smi, the referee): the deployed boot is
**21,436 used / 2,689 free** (ubatch 2048) — the pricing boots were the
MiMo-V2.6-Distill-Qwen-9B pair at this exact geometry (**21,276 / 2,849** at
ubatch 1024, **21,846 / 2,279** at 2048), so the 415 MiB lighter weights
bought back what the projection predicted. The engine ledger confirms the KV
law to the MiB: `CUDA0 KV buffer size = 12544.00 MiB` (`K (f16): 6272.00`,
`V (f16): 6272.00`), compute buffers 1,136 MiB on CUDA0 + 848 MiB host, the
mmproj 879 MiB + 248 MiB. Decode is ubatch-invariant (Swift-era 3090 law). The
Q6_K rung costs ~10 % decode vs Q5_K_M (weight bandwidth; MiMo-measured 94.4
vs 104.8 t/s).
**Vision**: the publisher's own BF16 mmproj via `--mmproj` (the model is
natively multimodal; the Q6_K file carries no vision tensors). Probe traps
**Vision**: the publisher's own Q8_0 mmproj via `--mmproj` (Qwen3-VL tower: 27
blocks, 768 px, patch 16, merge 2, `qwen3vl_merger`, deepstack). Probe traps
carry over: ≥256 px probe images or the model answers "video", and only the
OpenAI `image_url` content part reaches the model.
OpenAI `image_url` content part reaches the model. **New trap:** thinking
tokens count against `max_tokens`, so a vision probe at 64 tokens returns
EMPTY `content` with `finish_reason: "length"` — budget ≥512 or the endpoint
looks blind while it is only mid-thought.
**Base alignment**: Bonsai-2 is the censored BASE lineage (the 9B it replaced
was uncensored). The CRACK re-export on disk
(`/home/dank/models/Bonsai2-CRACK/`, header census proves byte-identical
geometry) is a one-line `MODEL_FILE` swap if a refusal ever surfaces.
**Compaction hub**: `models.yml` routes `compactionModel` for
`frank-archon`, `frank-dragoon` and `frank-zealot` (the 3090 endpoint) here — 200,704 is the
only window in the fleet that fits a worst-case compaction input (~170k
tokens). With `--parallel 2` a compaction request no longer serializes behind
a live generation (the Gemma4 era's `--parallel 1` law).
a live generation (the Gemma4 era's `--parallel 1` law). Retrieval through a
25,960-token prompt was verified exact after the swap.
**Contract law learned at verbosity 4:** a raw `/completion` request prints
`failed to parse reasoning content (… parse_reasoning=true)` whenever the
model closes a bare prompt with `` — cosmetic
(`probe_request.cpp:789`); the `/v1/chat/completions` path parses
`reasoning_content` correctly, which is the only path consumers use. Do not
lower `VERBOSITY` to silence it: the KV/buffer ledger is what VRAM forensics
read.
Runbook:
`serverAppConfigs-docs/LLM/frank-zealot-llama-qwen35-9b-q6k-2x200704.md`.
Rollback: revert the swap commit — the Gemma4 payload
`serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai2-pq20.md`.
Rollback: revert the swap commit — the Qwen3.5-9B Q6_K payload
(`/home/dank/models/Qwen35-9B-HauhauCS-Q6K/`) with the master CUDA build
(`~/GitPackages/llama-master/build-cuda/`), the Gemma4 payload
(`/home/dank/models/Gemma4-26B-A4B/`), the MiMo Q6_K payload
(`/home/dank/models/MiMo-Q6K/`), the Swift payload
(`/home/dank/models/Swift15-UncIQ3XXS/`) and the Ornith Q8_0 payload
(`/home/dank/models/Ornith-Q8/`) all remain on disk.
### frank-archon — Radiance (StillDeadcode), Swift1.5-Qwen3.8-Flash-Next GPTQ w4, TP=2 (Dankserver, ACTIVE)
### frank-archon — Radiance (StillDeadcode), publisher Qwen3.8-Flash-Next iq4r, TP=2 (Dankserver, ACTIVE)
The **radiance engine** (StillDeadcode, Apache-2.0; image
`stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned, **pin 1.0.13**
for serve/convert/KLD; `libr4d` gfx1201 kernels, plain `podman run`, root
systemd unit, container name = unit name) serving
**ukisai/Swift1.5-Qwen3.8-Flash-Next (BF16, 335.3 GiB, 131 shards,
397/397 sha256-verified)** converted by `scripts/convert-swift15-w4dn8.sh`
with recipe `scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe` to
`qwen4exp-swift15-w4dn8-i8-hc8m.rad` (~114 GiB): **all 49,152 experts GPTQ
w4** (act-order, damp 0.05, 10,000,000-token calibration captured on engine
**1.2.0** via `RADIANCE_CALIB_DIR`/`RADIANCE_CALIB_TOKENS` — the only pin
with the token knob; the RADGRAM2 grams are format-identical across pins and
cross the boundary by design), **24 whole experts per layer cast bf16** (the
PROTECTED set, derived from the calibration grams' energy share),
**int8 trunk/lm_head**, **fp8 HC mixtures** (`--hc-mix-dtype`), PLE n-gram
table at fp8 fixed scale (47.7 GiB in the container, from 95.4 GiB bf16).
The plugin's **one-table-one-encoding law** (measured at rad-convert
declare): one expert table takes ONE encoding; only whole bf16 experts are
legal beside it — no w4/fp8 split of the down/gate_up pair (24-value
`rad_arch_quant_desc` array, one per layer). KLD gate vs the BF16
reference (recorded on the same 1.0.13 pin) is the precision referee;
the throughput gate (decode ≥ 30.8 t/s) was passed at 4.5× the bar
(122–173 t/s at 44.5k depth). Measured decode/prefill in the runbook.
Rollback: revert the swap commit — the fp8-native `.rad`
(`qwen4exp-fp8-native.rad`) and every prior payload remain on disk.
for serve; `libr4d` gfx1201 kernels, plain `podman run`, root
systemd unit, container name = unit name) serving the
**publisher's container of the official Qwen/Qwen3.8-Flash-Next**:
`StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe` →
`qwen3.8-next-flash-fp8-iq4r-moe.rad` (116,322 MiB, Hub-LFS sha256
verified at fetch by `scripts/download-frank-archon-iq4r.sh`; recipe
`qwen4exp-w4nl64-i8-hc8m`): **routed experts as 4-bit codes into a
16-level non-uniform codebook** after a 128-point Walsh–Hadamard rotation
(E4M3 scale per 64 weights; codes by GPTQ on a 10M-token calibration with
per-expert down-proj Hessians, refined by 3 coordinate-descent sweeps),
**10 whole bf16-protected experts/layer**, **int8 W8A8 trunk/lm_head**
(scale per 128 columns searched), **fp8 HC mixtures**, PLE n-gram table at
fp8 fixed scale (47.7 GiB), **MTP head and bf16 vision tower in the
container**. Card quality (full-vocab teacher-forced KL vs bf16, ~75K
positions): mean 0.0789 all-tokens / 0.0239 assistant turns, top-1 92.85 %.
**MODEL_FILE is the only serving argument that moved** (2026-10-15); every
number of the 2026-10-13 shape below carries verbatim. The plugin's
**one-table-one-encoding law** (fleet-measured at rad-convert declare):
one expert table takes ONE encoding; only whole bf16 experts are legal
beside it. Rollback: revert the swap commit AND re-download the in-house
w4dn8 `.rad` (KLD-gated Swift1.5 conversion) — it is **no longer on disk**
(purged 2026-10-17); the byte-exact copy is published as
`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`
(LFS sha256 `09dd9306…`). The fp8-native `.rad` was deleted 2026-10-15
(rebuild: `scripts/download-frank-archon.sh` +
`scripts/build-frank-archon-rad.sh`).
**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 2`,
**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 3`,
`--max-model-len 200000` (the harness contextWindow must quote this),
`--placement expert_tiered --host-pool-mib 44928` + **explicit VRAM
partition `--vram-weights-mib 22208 --vram-kv-mib 6912`** (the vram-pair
partition `--vram-weights-mib 25600 --vram-kv-mib 5120`** (the vram-pair
replaced `--expert-vs-cache-ratio`; the two are ONE budget —
`weights + kv ≤ 29,120 MiB`, from the measured card 31,856 minus 1,710
consumed outside both budgets minus 1,024 to keep free), `--kv-cache-dtype`
**`bf16` — unquantized KV since 2026-10-11**, `--num-speculative-tokens 2`,
`--max-num-batched-tokens 2048`, prefix cache 8 GiB pinned host + 64 GiB
disk at `/home/dank/radiance-state`, **`--ngram-placement ram`**, Froggeric
v22.5 chat template vendored (`scripts/qwen38-froggeric-v22.5.jinja`,
`--override-chat-template`). Boot ledger: KV pool 6.75 GiB = **462,752 bf16
tokens, "backs 2 sequences all at 200000 tokens at once"** (the fp8 pool's
386,904 backed 1.93), weights slab filled to 147 KiB of slack, host pool
10,375 slots, cards settled 1.85 GiB free.
`weights + kv ≤ 30,969 MiB`, derived from the engine's own budget table: card
32,624 MiB, the pair at 29,120 settling 2,058 MiB free against the engine's
209.19 MiB floor (`--gpu-headroom-mib 96` + 113.19 static), so 1,849 MiB is
claimable and the old 29,120 bound was my hand-picked 1 GiB reserve, not a
hardware wall), `--kv-cache-dtype` **`fp8` — quantized KV, restored 2026-10-13
because it is the only dtype that funds the slot count at this window**,
`--num-speculative-tokens 2`, `--max-num-batched-tokens 2048`, prefix cache
8 GiB pinned host + 64 GiB disk at `/home/dank/radiance-state`,
**`--ngram-placement ram`**, Froggeric v22.5 chat template vendored
(`scripts/qwen38-froggeric-v22.5.jinja`, `--override-chat-template`). Boot
ledger (2026-10-13, pair 25600/5120): `kv cache 5.00 GiB … backs 3 sequences
all at 200000 tokens at once, which is the worst case` — `kv_attn` page
26.00 KiB (fp8) at 607,740 tokens (the sizing law predicted 608,027),
`kv_qsa_bkey` 3.25 KiB at the same token count, `kv_gdn_state` 54 MiB × 6 slots
(3 sequences × 2 copies) = 324 MiB; the sampler prints **9 rows**
(3 × (1+2), inside the 25-row `wht6` exactness bound); the slab filled to
563 KiB of slack with **17,548 slots** (16,235 at 24,000, 14,753 at 22,208 —
every 1,792 MiB buys ~1,300 resident experts, and the pinned host pool hands
the same experts back: 8,893 → 7,580 slots); `unclaimed 477.30 MiB`, cards
settled ~336 MiB free against the engine's 209.19 MiB floor, KV lending
4.22 GiB to the expert cache. Prefill tracks this slab only up to a
**threshold**: 22,208 → 3,397 and 24,000 → 3,290 t/s at 43.5k are one band
(±3 %), and the 25,600 arm measured **3,290** — the third step bought nothing
(the 3,584 arm did not reproduce at 3 slots); the cliff is at ~26 GiB
(26,071 → 4,379 on 2026-10-11's arm), so below that the slab is a threshold,
not a slope.
**Decode ladder (2026-10-13, 512 tokens, greedy, this shape)**: 99.0 t/s
single-stream at 43.5k, 93.3 at 197k — decode is nearly window-free (−6 % for
4.5× depth); **3 concurrent streams give 113.9/43.3/10.9 t/s = 18.4 t/s
aggregate**, so the slot count is paid in decode concurrency (a fused-step
lottery), not in prefill; leg D of `probe-frank-archon-kv.py` measures it.
Shape notes carried from the fp8-native era and unchanged: wht6 gains,
MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk
@@ -551,35 +597,53 @@ on the fp8 shape (512-token streamed probe). The campaign also
measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth,
accept 0.7246).
**The bf16 KV law (2026-10-11)**: `--kv-cache-dtype` takes ONLY `bf16|fp8` —
one dtype for the whole cache, so llama.cpp's exact-K/quantized-V pair (what
frank-zealot runs) has NO radiance equivalent; the rung under bf16 is fp8 on
both halves. Measured per rank from the two boot ledgers: `kv_attn` (13
layers, 4-token blocks) pages 26.00 → 52.00 KiB, while `kv_qsa_bkey` (13
layers) stays 3.25 KiB at BOTH dtypes (the QSA block key is an index, not
attention state) — so a token costs 7.3125 KiB at fp8 and 13.8125 KiB at
bf16: **1.89×, not 2×**. Pool sizing must also pay the dtype-independent terms
that live in the same pool (~110 MiB a sequence of recurrent state + 448 MiB
of prefix checkpoints): `(vram_kv − 110·seqs − 448) MiB / 13.8125 KiB` =
462,946 where the engine reported 462,752 (0.04 % page rounding). The cost is
paid twice on this engine, and half-refunded: idle KV VRAM is also the expert
cache's spillover, and the bigger pool *lends* 6.08 GiB to the slab where the
fp8 pool lent 2.69 GiB. `--max-num-seqs 3 → 2` came with the change (two slots
× the window is the capacity contract; the "seqs 2 cannot boot" law belongs to
the retired tcclaviger fork's torch.compile, not radiance) and it also drops
the fused all-reduce to 2 × 3 = 6 rows, under the 25-row bound where wht6 is
bit-exact. Disk-tier sizing law: the engine forces ~9.6 GiB of disk-tier weight
buffers + arena onto the weights side, so `weights + kv` is capped near
29.6 GiB/card; boot with the pair, never `--kv-cache-size-mib`.
**The KV dtype law (measured 2026-10-11, applied at 4 slots 2026-10-13)**:
`--kv-cache-dtype` takes ONLY `bf16|fp8` — one dtype for the whole cache, so
llama.cpp's exact-K/quantized-V pair (the 9B-era frank-zealot ran it; the
Bonsai-2 swap runs q8_0/q8_0) has NO radiance
equivalent; the rung under bf16 is fp8 on both halves. Measured per rank from
the two boot ledgers: `kv_attn` (13 layers, 4-token blocks) pages 26.00 → 52.00
KiB, while `kv_qsa_bkey` (13 layers) stays 3.25 KiB at BOTH dtypes (the QSA
block key is an index, not attention state) — so a token costs 7.3125 KiB at fp8
and 13.8125 KiB at bf16: **1.89×, not 2×**. Pool sizing must also pay the
dtype-independent terms that live in the same pool (~110 MiB a sequence of
recurrent state + 448 MiB of prefix checkpoints):
`(vram_kv − 110·seqs − 448) MiB / rate` — bf16 at 6,912 gives 462,946 where the
engine reported 462,752 (0.04 % page rounding); fp8 at 6,912 with four slots
gives 843,566, i.e. the deployed capacity contract 4 × 200,000. **That is why
the dtype is fp8: the pool must back `seqs × max-model-len` as a hard limit, and
bf16 at four slots costs 11,679 MiB — the slab would fall to 17,441 MiB (−33 %),
where fp8 leaves it at 22,208.** The cost is paid twice on this engine, and
half-refunded: idle KV VRAM is also the expert cache's spillover, and the bigger
pool *lends* 6.08 GiB to the slab where the 3,543 MiB fp8 pool lent 2.69 GiB.
`--max-num-seqs` rides 4 × (1 + spec) = 12 fused-all-reduce rows, under the
25-row bound where wht6 is bit-exact (the "seqs 2 cannot boot" law belongs to
the retired tcclaviger fork's torch.compile, not radiance). Disk-tier sizing
law: the engine forces ~9.6 GiB of disk-tier weight buffers + arena onto the
weights side, so `weights + kv` is capped near 29.6 GiB/card; boot with the
pair, never `--kv-cache-size-mib`.
Shape notes carried from the fp8-native era and unchanged: wht6 gains,
MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk
prefix cache, and the rule that a prompt over `--max-model-len` is
refused, never truncated. What changed: the fp8-native era's NVMe expert
streaming (the 3.5 GB/s wall) is gone — the w4 resident set fits the
host pool, so decode ran 122–173 t/s in the campaign arms and ~190 t/s
on the fp8 shape (512-token streamed probe). The campaign also
measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth,
accept 0.7246).
**The pinned-RAM ledger is why this is a root unit**: 2 × 44 GiB expert
pools + the 47.7 GiB n-gram copy (`--ngram-placement ram` is pinned RAM
the card reads) + 8 GiB prefix host ≈ **147 GiB** charged against
`RLIMIT_MEMLOCK` — the TTM floor is set just above it.
`ttm.pages_limit`/`pages_max` cap GPU-mappable pinned pages; the serve
script's TTM preflight (`FRANK_ARCHON_TTM_MIN_MIB=150000`, cap on this host
204,800) refuses to start until both knobs are raised in the stop window
(sysfs write first, kernel cmdline + reboot only if read-only).
`ttm.pages_limit`/`pages_max` cap GPU-mappable pinned pages, and **the knob is
runtime state whose default is half of RAM**: `scripts/ttm-preflight.sh` (sourced
by the serve script) therefore RAISES it to `FRANK_ARCHON_TTM_TARGET_MIB=204800`
before refusing, because the 2026-10-12 power failure reset it to 125,988 MiB
and the preflight re-refused 6,919 times. A manual sysfs write in the stop
window is a convenience, not the repair; kernel cmdline persistence
(`ttm.pages_limit=52428800`) is the belt to it.
**Contract facts** (source-verified): the HTTP listener binds AFTER the
full load, so a bare `curl /health` gate is honest (`/health` == `/ping`,
@@ -591,8 +655,9 @@ chat format — no parser flags; vision native (the 27-layer Qwen-VL tower,
`--mm-max-patches`); `/metrics` carries vLLM-compatible names +
`radiance:*` families — the Dankclaw Prometheus scrape is re-added.
Runbook: `serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md`.
Retired fp8-native era: `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-native-fp8.md`.
Runbook: `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-iq4r.md`.
Retired Swift1.5 w4dn8 era (payload purged 2026-10-17; rollback = HF re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`): `serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md`.
Retired fp8-native era (payload deleted 2026-10-15, rebuildable): `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-native-fp8.md`.
Retired FreeToken era: `serverAppConfigs-docs/LLM/frank-archon-freetoken-pr587.md`.
Retired vLLM era: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-archon-glm53-flash-w4a16.md`.
@@ -761,6 +826,17 @@ linked runbooks.
the KV pool is worse than the OOM it prevents.
- Host swap state is invisible to `deploy.sh` (no repo file) — check
`swapon --show` before reasoning about headroom.
- **A sysfs raise is a countdown to the next reboot.** `ttm.pages_limit` defaults
to half of RAM, so the manual raise to 200 GiB (2026-10-05) vanished with the
2026-10-12 23:17 POWER FAILURE on dankserver: the cap fell to 125,988 MiB,
frank-archon's pinned-ledger preflight (floor 150,000 MiB) refused the start,
and `Restart=always` re-ran that refusal **6,919 times in ~20 h** — a unit in
`activating (auto-restart)` that exits 1 in 50 ms with no container and no
engine log is a PREFLIGHT refusal, not an engine fault. Repo law now:
`scripts/ttm-preflight.sh` (sourced by `serve-frank-archon.sh`) raises the knob
to `FRANK_ARCHON_TTM_TARGET_MIB` itself, re-reads, and refuses by name only
after a write the kernel rejected. Never prescribe a manual sysfs write as the
recovery procedure for a service that can boot before a human reaches it.
- **Silent RAM corruption outranks every software theory.** frank-archon's
~40k-token output corruption (2026-10-04/05, repetition/garbage in long
agent conversations) was marginal DDR5 at 4800 MT/s on four DIMMs, fixed
@@ -969,6 +1045,31 @@ linked runbooks.
`Unexpected reasoning effort high` until `high: xhigh` was added on
2026-09-16. Halogen accepts the wider set, so the two entries legitimately
differ.
- **The vision projector is NOT billed in llama.cpp's own fit check.** The
engine prints `[mtmd] estimated worst-case memory usage of mmproj is
873.10 MiB` and then fits WITHOUT it: frank-zealot's Bonsai-2 boot projected
21,439 MiB and settled at 22,664 (2026-10-17). Price a 3090-class shape from
settled `nvidia-smi` after a real boot, never from
`model + context + compute` — the projector (and any drafter context) lands
after `/health` would already have answered. Same failure class as the
DSpark drafter OOM.
- **The FIRST request after a boot is a warmup artifact.** Measured on
Bonsai-2/3090: 2.6 t/s prefill and 37.7 t/s decode on request #1, then
1,258 t/s prefill and 54.7 t/s (per stream, two concurrent) on the next — a
3× spread on the same shape. Benchmark the second request or later, or the
graph-capture cost reads as a model fault.
- **Thinking tokens count against `max_tokens`, so a short-budget vision probe
reads as blindness.** The Bonsai-2/`image_url` probe at `max_tokens: 64`
returned `content: ""` with `finish_reason: "length"` and half a thought in
`reasoning_content` — the image HAD reached the model (322 prompt tokens vs
42 for the text alone). At 512 tokens the same image answers "Red". Budget
≥512 for any thinking model you probe for vision.
- **Quantized KV is not the only dtype law a fork build can hide.** Vendor
builds carry their own compile options: frank-zealot's PrismML prebuilt
served `--cache-type-v q8_0` under flash attention at the exact 34,816
B/token (the `GGML_CUDA_FA_ALL_QUANTS` door), so a build decision must be
proven by the KV buffer line in the boot ledger, not by the build flags you
remember passing.
**Ops**
@@ -1040,8 +1141,8 @@ Active:
| Service | Runbook |
|---------|---------|
| frank-dragoon (flash-server 0.16.2, abliterated Swift1.5 v2 .hgn + MTP always-on, 2×262,144, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-swift15.md` |
| frank-zealot (llama.cpp CUDA, Qwen3.5-9B-Uncensored Q6_K, 2×200,704 exact f16 KV MTP-off, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-zealot-llama-qwen35-9b-q6k-2x200704.md` |
| frank-archon (radiance StillDeadcode, Qwen3.8-Flash-Next-FP8 native, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-native-fp8.md` |
| frank-zealot (llama.cpp PrismML fork, Bonsai-2-27B PQ2_0, 2×200,704 q8_0 KV MTP-off, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai2-pq20.md` |
| frank-archon (radiance StillDeadcode, publisher Qwen3.8-Flash-Next iq4r .rad, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-iq4r.md` |
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
@@ -1049,6 +1150,8 @@ Retired (rollback targets and history):
| Runtime | Host | Window | Runbook |
|---------|------|--------|---------|
| radiance Swift1.5-Qwen3.8-Flash-Next GPTQ w4dn8, 3×200,000 fp8-KV slab 25600 (`frank-archon`) | Dankserver | 2026-10-08 → 10-15 (rollback target — `.rad` purged 2026-10-17, byte-exact re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`; owns the slab-slope ladder, KV dtype law, KLD campaign, and every shape law the iq4r swap carried verbatim) | `frank-archon-radiance-swift15-w4dn8.md` |
| llama.cpp CUDA (master) Qwen3.5-9B-Uncensored Q6_K, 2×200,704 exact f16 KV MTP-off (`frank-zealot`) | Dankclaw | 2026-10-07 → 10-17 (rollback target — payload + master CUDA build on disk; owns the 32 KiB/token 8-full-layer KV law and the 2→3-slot reslice law) | `frank-zealot-llama-qwen35-9b-q6k-2x200704.md` |
| llama.cpp CUDA Gemma4-26B-A4B QAT Q4_K_M, 1×200,704 f16-K/q8_0-V MTP-off (`frank-zealot`) | Dankclaw | 2026-10-07 → 10-07 (same-day swap; rollback target — payload on disk; owns the `attention_k_eq_v` pair-billing KV law and the gemma4-assistant drafter-cost boots) | `frank-zealot-llama-gemma4-q4km-200704.md` |
| ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-dragoon`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-dragoon-rocmfpx-q6agent-uncensored-mtp.md` |
| tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-archon`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-archon-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` |
+11
View File
@@ -1,5 +1,16 @@
# Server App Configs
### AI Disclosures:
- Slop rating: 2/5 - Actively using, actively de-slopping, **repository mostly human owned**, lacks formal process.
- Completeness: Unfinished, planning planned, no plan yet
- Datacenter model/spend: Less than $20 of Deepseek 4.1 Flash and GLM-5.3-Flash
- Local model: Qwen 3.8 Flash Next
### slop rating of 2 or lower indicates human reading is a worthwhile exercise, and that Frank has read the code and knows the architecture.
### slop rating of 2 indicates that while this project is human owned, no human contribution process exists and/or release is agentically driven.
The single repository for the two homelab servers: the Podman container
stacks **and** the LLM inference fleet. One repo, one checkout per host,
one deploy script.
+28 -22
View File
@@ -1222,23 +1222,26 @@ deploy_llm_dragoon() {
wait_for_health frank-dragoon http://127.0.0.1:7770/health 1200
}
# deploy_llm_zealot — frank-zealot is llama.cpp (master build, CUDA sm_86) on
# the RTX 3090 as a ROOT unit: the runtime is a native binary tree under
# ~/GitPackages/llama-master/build-cuda, so there is no container image and
# no generated unit to avoid. relink_unit points
# deploy_llm_zealot — frank-zealot is llama.cpp (PrismML fork build) on the
# RTX 3090 as a ROOT unit: the runtime is a native binary tree under
# ~/GitPackages/llama-prism (vendor release build 10754, commit 2459f68b5 —
# the ONLY loader that registers the payload's custom tensor type 142 and
# reads its prism.hadamard.* metadata), so there is no container image and no
# generated unit to avoid. relink_unit points
# /etc/systemd/system/frank-zealot.service at the repo file and boot
# persistence is that unit's [Install] section.
#
# It is a SEPARATE endpoint on port 7771; frank-dragoon keeps 7770. The 3090 is
# not shared: frank-dragoon's engine runs on the gfx1151 iGPU.
LLM_ZEALOT_MODEL_FILE="${LLM_ZEALOT_MODEL_FILE:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf}"
LLM_ZEALOT_RUNTIME="${LLM_ZEALOT_RUNTIME:-/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
# The projector IS the vision capability here: the model is natively
# multimodal and the Q6_K file carries no vision tensors — the publisher ships
# its own BF16 projector in the same repo, so a deploy that starts without
# this file silently ships a text-only endpoint. Checked unconditionally, like
# frank-dragoon's.
LLM_ZEALOT_MMPROJ="${LLM_ZEALOT_MMPROJ:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf}"
LLM_ZEALOT_MODEL_FILE="${LLM_ZEALOT_MODEL_FILE:-/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-PQ2_0.gguf}"
LLM_ZEALOT_RUNTIME="${LLM_ZEALOT_RUNTIME:-/home/dank/GitPackages/llama-prism/llama-server}"
# The projector IS the vision capability here: the Bonsai-2 text file carries
# no vision tensors, so the publisher's own mmproj (Qwen3-VL tower, 27 blocks,
# 768 px, qwen3vl_merger, 629,246,976 B) is the whole image path — a deploy
# that starts without it silently ships a text-only endpoint. Checked
# unconditionally, like frank-dragoon's. Its 873.10 MiB worst-case VRAM lands
# AFTER the engine's fit check, so it is not inside the q8_0 budget.
LLM_ZEALOT_MMPROJ="${LLM_ZEALOT_MMPROJ:-/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf}"
deploy_llm_zealot() {
# Refuse rather than restart-loop: a unit whose runtime or payload is
@@ -1246,15 +1249,15 @@ deploy_llm_zealot() {
# endpoint dark with nothing in the log to say why.
if [[ ! -x "$LLM_ZEALOT_RUNTIME" ]]; then
echo "ERROR: llama.cpp runtime absent at $LLM_ZEALOT_RUNTIME" >&2
echo "frank-zealot: build it (cmake -B build-cuda -DGGML_CUDA=ON" >&2
echo "frank-zealot: -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13" >&2
echo "frank-zealot: && cmake --build build-cuda --target llama-server) first." >&2
echo "frank-zealot: fetch the PrismML fork release build (build 10754," >&2
echo "frank-zealot: commit 2459f68b5) into ~/GitPackages/llama-prism first —" >&2
echo "frank-zealot: stock llama.cpp refuses the payload's custom tensor type 142." >&2
return 1
fi
for f in "$LLM_ZEALOT_MODEL_FILE" "$LLM_ZEALOT_MMPROJ"; do
if [[ ! -f "$f" ]]; then
echo "ERROR: frank-zealot payload file absent at $f" >&2
echo "frank-zealot: run scripts/download-frank-zealot.sh first;" >&2
echo "frank-zealot: run scripts/download-frank-zealot-bonsai2.sh first;" >&2
echo "frank-zealot: refusing to start a unit that cannot load." >&2
return 1
fi
@@ -1277,11 +1280,14 @@ deploy_llm_zealot() {
}
# deploy_llm_archon — frank-archon is the radiance engine (StillDeadcode,
# digest-pinned stilldeadcode/radiance image) serving the OFFICIAL
# Qwen/Qwen3.8-Flash-Next-FP8 at native block-FP8 precision from the
# rad-convert container qwen4exp-fp8-native.rad, on BOTH R9700s (TP=2) as a
# digest-pinned stilldeadcode/radiance image) serving the PUBLISHER's
# 4-bit-codebook container of the official Qwen/Qwen3.8-Flash-Next
# (qwen3.8-next-flash-fp8-iq4r-moe.rad, fetched by
# scripts/download-frank-archon-iq4r.sh) on BOTH R9700s (TP=2) as a
# ROOT unit whose serve script drives `podman run` directly (the engine is
# the image entrypoint; the image lives in dank's rootless podman storage).
# 2026-10-15 payload swap from the in-house w4dn8 (payload on disk =
# rollback).
# 2026-10-05 cutover from FreeToken (pre-cutover tag
# frank-archon-freetoken-pr587-20261005); before that, the tcclaviger vLLM
# fork (tag frank-archon-vllm-shapeG-20261003). Same port and alias; the
@@ -1290,7 +1296,7 @@ deploy_llm_zealot() {
# Gates: the image must exist under its digest ref in dank's rootless
# storage (FRANK_ARCHON_IMAGE), the .rad container file must exist at the
# env's size floor (FRANK_ARCHON_MODEL_FILE >= FRANK_ARCHON_RAD_MIN_MIB —
# a truncated convert is a guaranteed long-timeout boot), and the unit env
# a truncated fetch is a guaranteed long-timeout boot), and the unit env
# must be complete. Per-file sha256 verification is the fetch script's job
# at download time and rad-convert/rad-info's job at build time, never
# re-hashed here.
@@ -1325,13 +1331,13 @@ deploy_llm_archon() {
fi
if [[ ! -f "$model_dir/$model_file" ]]; then
echo "ERROR: .rad container missing: $model_dir/$model_file" >&2
echo "frank-archon: run scripts/build-frank-archon-rad.sh" >&2
echo "frank-archon: run scripts/download-frank-archon-iq4r.sh --fetch" >&2
return 1
fi
mib=$(( $(stat -c %s "$model_dir/$model_file") / 1048576 ))
if [[ "$mib" -lt "$rad_min" ]]; then
echo "ERROR: .rad container too small: $model_dir/$model_file is ${mib} MiB, want >= $rad_min" >&2
echo "frank-archon: rebuild it (scripts/build-frank-archon-rad.sh --force)" >&2
echo "frank-archon: re-fetch it (scripts/download-frank-archon-iq4r.sh --fetch; delete the staged file first if its sha256 failed)" >&2
return 1
fi
+83
View File
@@ -0,0 +1,83 @@
#!/usr/bin/env bash
# download-frank-archon-iq4r.sh — payload fetch/verify for the frank-archon
# swap to the PUBLISHER's 4-bit-codebook container:
# StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe — the official
# Qwen/Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's own
# author into ONE .rad: routed experts as 4-bit codes into a 16-level
# non-uniform codebook (Walsh–Hadamard rotation, GPTQ + coordinate-descent,
# 10M-token calibration), ten bf16-protected experts per layer, int8 trunk +
# lm_head, fp8 HC mixtures, the PLE n-gram table at fp8 fixed scale, the MTP
# head, and the bf16 vision tower. File:
# qwen3.8-next-flash-fp8-iq4r-moe.rad — 121,969,901,568 B (113.6 GiB,
# 116,322 MiB), card KL 0.0789 all-tokens / top-1 92.85 % vs bf16.
# Repo is UNGATED — no token needed (never put one on the command line).
#
# bash scripts/download-frank-archon-iq4r.sh --fetch resume/complete
# bash scripts/download-frank-archon-iq4r.sh verify only
#
# Fetch uses the vendored scripts/hfget.sh (curl+jq only): per-file size
# checks against the Hub listing, `-C -` resume, disk-fit gate. The side
# files (LICENSE, README, the w4nl recipe) stay in the staging dir as
# provenance; the serve gate mounts ONLY /home/dank/models/rad, so the
# verified .rad alone moves into it.
#
# Verify (default) is the AGENTS.md law in two parts: the file lands at the
# final path with the Hub's EXACT byte count, and its sha256 equals the
# Hub's LFS oid. A truncated transfer is a guaranteed long-timeout boot
# without this gate.
set -euo pipefail
REPO="StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe"
RAD_NAME="qwen3.8-next-flash-fp8-iq4r-moe.rad"
STAGE="/home/dank/models/StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe"
DEST="/home/dank/models/rad/$RAD_NAME"
SIZE=121969901568
SHA256="0af5e96244e80c21ac8edca1719dae49b8a201b94a2f3994c3e24026ceaa4d20"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
mode="verify"
[[ "${1:-}" == "--fetch" ]] && mode="fetch"
if [[ "$mode" == "fetch" ]]; then
# HFGET_JOBS=4 is hfget's default (a few streams saturate the link).
bash "$SCRIPT_DIR/hfget.sh" "$REPO" "$STAGE" main
staged="$STAGE/$RAD_NAME"
if [[ ! -f "$staged" ]]; then
echo "FETCH FAILED: $staged absent after hfget" >&2
exit 1
fi
got=$(stat -c %s "$staged")
if [[ "$got" != "$SIZE" ]]; then
echo "FETCH FAILED: staged .rad is $got B, want $SIZE B" >&2
exit 1
fi
echo "staged .rad byte-exact ($SIZE B); hashing (one full read, minutes)"
got_sha=$(sha256sum "$staged" | cut -d' ' -f1)
if [[ "$got_sha" != "$SHA256" ]]; then
echo "FETCH FAILED: sha256 $got_sha, want $SHA256 — DELETE the staged" \
"file before re-fetching (curl -C - cannot repair a corrupt body)" >&2
exit 1
fi
mkdir -p "$(dirname "$DEST")"
mv "$staged" "$DEST"
echo "payload placed: $DEST"
fi
echo "verifying deployed payload: $DEST"
if [[ ! -f "$DEST" ]]; then
echo " BAD payload absent: $DEST" >&2
exit 1
fi
got=$(stat -c %s "$DEST")
if [[ "$got" != "$SIZE" ]]; then
echo " BAD payload is $got B, want $SIZE B" >&2
exit 1
fi
echo "byte-exact ($SIZE B); hashing (one full read, minutes)"
got_sha=$(sha256sum "$DEST" | cut -d' ' -f1)
if [[ "$got_sha" != "$SHA256" ]]; then
echo " BAD sha256 $got_sha, want $SHA256" >&2
exit 1
fi
echo "payload verified: $RAD_NAME, size and sha256 match the Hub LFS oid"
+93
View File
@@ -0,0 +1,93 @@
#!/usr/bin/env bash
# Fetch candidate payload for frank-zealot: prism-ml/Ternary-Bonsai-2-27B-gguf
# at **PQ2_0** (the ternary g128 pack stored 2 bits/trit + fp16 scale per 128)
# plus the publisher's Q8_0 vision projector. Public repo — no auth header,
# and never a token on the command line (world-readable in /proc/<pid>/cmdline).
#
# Why PQ2_0 and not PTQ1_0: measured on the same silicon family (community
# 3090 Ti report, PrismML-Eng/Bonsai-demo) PTQ1_0 is ~17 % slower decode and
# ~48 % slower prefill than PQ2_0 on Ampere; the model card says the same
# (PTQ1_0 wins only on Ada/L4). The pack is consumed packed — 6.71 GiB
# resident, never expanded to f16.
#
# Header facts (HTTP range parse, not the card): arch qwen35, 64 blocks,
# full_attention_interval 4 => 16 full-attention + 48 Gated DeltaNet layers,
# 4 KV heads x 256 dim, native 262,144, vocab 248,320, and NO nextn/mtp
# tensors (block_count is 64 real layers) — so in-file MTP is impossible and
# SPEC_TYPE=none is forced by the payload. 402 tensors carry the custom type
# id 142 (GGML_TYPE_PQ2_0, group 128): STOCK llama.cpp refuses this file, and
# loads the repo's F16 file silently with garbage output — only the
# PrismML-Eng/llama.cpp fork applies the prism.hadamard.* activation
# transform declared in this file's metadata.
#
# These ARE LFS files (unlike the HauhauCS payload): the Hub publishes the
# oid, and both shas below are the Hub's LFS sha256. The mmproj sha has also
# been recomputed locally against the copy in /home/dank/models/Bonsai2-CRACK/.
#
# NOT pinned by this script: /home/dank/models/Bonsai2-CRACK/
# Bonsai-2-27B-PQ2_0-CRACK.gguf (7,206,168,928 B, sha256 5b24ea3eebc3e0bc...,
# fetched 2026-10-06 by an undocumented ad-hoc path). Same size as the
# official file, different bytes — a dealign/CRACK (uncensored) variant of the
# same ternary pack. Provenance unverified; it is a candidate arm, not this
# script's payload.
#
# Usage:
# scripts/download-frank-zealot-bonsai2.sh [DEST_DIR] [--verify-only]
set -euo pipefail
DEST=""
VERIFY_ONLY=0
for arg in "$@"; do
case "$arg" in
--verify-only) VERIFY_ONLY=1 ;;
*) DEST="$arg" ;;
esac
done
DEST="${DEST:-/home/dank/models/Bonsai2-PQ2_0}"
# repo | hub-name | dest-name | bytes | sha256 (Hub LFS oid)
FILES=(
"prism-ml/Ternary-Bonsai-2-27B-gguf|Ternary-Bonsai-2-27B-PQ2_0.gguf|Ternary-Bonsai-2-27B-PQ2_0.gguf|7206168928|3907dc1658db1f78a9826bf8d5bcb8dc65db0d466388937af57f2294fae62ec1"
"prism-ml/Ternary-Bonsai-2-27B-gguf|Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf|Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf|629246976|6807ede61d570bb86ba34b756a0fa109edc33668604de867c6ea6d8f1d631903"
)
mkdir -p "$DEST"
status=0
for entry in "${FILES[@]}"; do
IFS='|' read -r repo hubname destname size sha <<<"$entry"
url="https://huggingface.co/${repo}/resolve/main/${hubname}"
f="$DEST/$destname"
if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]] \
&& sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
echo "OK $destname"
continue
fi
if (( VERIFY_ONLY )); then
echo "MISSING $destname"
status=1
continue
fi
echo "FETCHING $destname (from $repo)"
tmp="$f.part"
# No resume: `curl -C -` once produced an oversized file on a 31 GiB
# payload. -f/--fail-with-body so an HTTP 401/403 body is never a success.
curl -fsSL --fail-with-body --retry 3 --retry-delay 5 -o "$tmp" "$url"
mv "$tmp" "$f"
if [[ "$(stat -c%s "$f")" != "$size" ]]; then
echo "SIZE FAIL $destname ($(stat -c%s "$f") != $size)"
status=1
continue
fi
if ! sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
echo "SHA FAIL $destname"
status=1
continue
fi
echo "VERIFIED $destname"
done
if (( status == 0 )); then
echo "DOWNLOAD_DONE $DEST"
else
echo "DOWNLOAD_FAILED $DEST" >&2
fi
exit "$status"
+155 -77
View File
@@ -1,12 +1,31 @@
# frank-archon.env — radiance (StillDeadcode, Apache-2.0) serving
# ukisai/Swift1.5-Qwen3.8-Flash-Next converted to GPTQ w4 (MODEL_FILE below,
# built by scripts/convert-swift15-w4dn8.sh) on BOTH R9700s (TP=2),
# dankserver :7770.
# frank-archon.env — radiance (StillDeadcode, Apache-2.0) serving the
# PUBLISHER's 4-bit-codebook container of the official Qwen/Qwen3.8-Flash-Next
# (MODEL_FILE below, fetched by scripts/download-frank-archon-iq4r.sh) on
# BOTH R9700s (TP=2), dankserver :7770.
# 2026-10-15 PAYLOAD SWAP: the model moved from the in-house w4dn8 conversion
# of the Swift1.5 finetune to the engine author's published container
# qwen3.8-next-flash-fp8-iq4r-moe.rad (116,322 MiB, sha256-verified at fetch —
# the w4dn8 measured 116,416 MiB, so this shape carries over with 94 MiB to
# spare). NO argument changed: every number below is the 2026-10-13 shape,
# kept verbatim. The swap commit is the rollback (the w4dn8 payload stays on
# disk).
#
# SHAPE (2026-10-11): 2 decode slots x 200,000 tokens with an UNQUANTIZED
# (bf16) KV cache — the pool is sized for the pair, and the weights slab pays
# for it (see the KV block). The fp8-native payload this file was written for
# is the rollback target and keeps its own trio under the fp8-native runbook.
# SHAPE (2026-10-13): THREE decode slots x 200,000 tokens, fp8 (quantized) KV,
# with the per-card budget spent on resident experts: 25,600 MiB of slab +
# 5,120 MiB of pool = 30,720 MiB, inside the 30,969 MiB ceiling the engine's own
# budget table allows (card 32,624 MiB, settle floor 209 MiB). The pool is sized
# to the contract and nothing more (3 x 200,000 needs 5,063 MiB at fp8; 5,120
# predicts 608,027 tokens, 1.3 % slack), because every MiB taken from the slab
# is prefill and TTFT (2026-10-11 A/B: slab -14.8 % cost prefill -29 %, TTFT
# +25 % at the full window) while KV beyond the contract buys nothing.
# Consequence, deliberate: a FOURTH full-height conversation does not fit this
# pair, and the engine refuses it — it never truncates. See the KV block for the
# arithmetic and the measured dtype law. The w4dn8 payload is the swap
# rollback and keeps its trio under the w4dn8 runbook; the fp8-native payload
# that came before it was DELETED from disk 2026-10-15 (Frank's call, to free
# the volume for this download — rebuildable via the fetch + build scripts
# named in the fp8-native runbook).
#
# 2026-10-05 cutover from FreeToken (PR #587 ROCm, one card, experts in
# pinned host RAM; git tag frank-archon-freetoken-pr587-20261005 is the
@@ -21,20 +40,30 @@
# (`X.weight_scale_inv` is the scale plane — checkpoint.h). The PLE n-gram
# table (51B params, 47.7 GiB) is served from pinned host RAM via io_uring
# reads — Frank's call: "Native FP8. Experts from ram. Pin the PLE to ram."
# The 4-bit-codebook publisher shape (qwen3.8-next-flash-fp8-iq4r-moe.rad,
# --host-pool-mib 12288) remains the documented fallback if the native
# container proves unbootable.
# THE 4-bit-codebook publisher shape is the DEPLOYED payload since
# 2026-10-15. The card's own run line (--host-pool-mib 12288,
# --expert-vs-cache-ratio 0.82, seqs 8, spec 3) is the publisher's
# suggestion on 32 GB cards; this file's shape is the fleet's measured one.
#
# PAYLOAD: /home/dank/models/rad/qwen4exp-fp8-native.rad — a rad-convert
# NO-RECIPE pure copy of the official 131-shard checkpoint (every weight
# keeps its block-FP8 encoding; ~161 GiB). The convert runs against a
# staging dir whose config.json drops `quantization_config`: the descriptor
# string is only the plugin SELECTION key (spec §2.4 — exact match, no
# fallback), and no shipped plugin claims (qwen4exp, "fp8"); the shipped
# qwen4exp plugin claims (qwen4exp, "") which is what recipe-built
# containers carry. Per-weight encodings are data, not the descriptor —
# removing the key changes nothing about what the weights hold. Built by
# scripts/build-frank-archon-rad.sh (staging + convert + rad-info).
# PAYLOAD: /home/dank/models/rad/qwen3.8-next-flash-fp8-iq4r-moe.rad — the
# official Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's
# own author (recipe qwen4exp-w4nl64-i8-hc8m: experts as 4-bit codes into a
# 16-level non-uniform codebook after a 128-point Walsh–Hadamard rotation,
# GPTQ + 3 coordinate-descent sweeps on a 10M-token calibration, ten
# bf16-protected experts per layer, int8 W8A8 trunk + lm_head, fp8 HC
# mixtures, PLE n-gram fp8 fixed-scale, MTP head, bf16 vision tower; card KL
# 0.0789 all-tokens / top-1 92.85 % vs bf16). Fetched byte-exact + sha256 by
# scripts/download-frank-archon-iq4r.sh; the recipe/LICENSE stay in the
# staging dir beside it as provenance. ROLLBACK: revert the swap commit — the
# in-house qwen4exp-swift15-w4dn8-i8-hc8m.rad (scripts/convert-swift15-w4dn8.sh,
# KLD-gated) stays on disk. Behind it, the fp8-native
# qwen4exp-fp8-native.rad was DELETED from disk 2026-10-15 (Frank's call,
# one-time rm, to free the volume for this payload); rebuild =
# scripts/download-frank-archon.sh + scripts/build-frank-archon-rad.sh — a
# NO-RECIPE rad-convert copy of the official 131-shard checkpoint under the
# descriptor law (no shipped plugin claims (qwen4exp, "fp8"), so the staging
# config.json drops quantization_config; per-weight encodings are data, not
# the descriptor).
#
# SERVING FACTS (docs/GUIDE.md, docs/DOCKER.md, compose/flashnext.yaml):
# - The HTTP listener binds AFTER the full load (main.cpp: engine.start()
@@ -50,12 +79,12 @@
# enable_thinking, preserve_thinking (core/server/oai.cpp:194-205) —
# models.yml's extraBody set is fully inside it. Unknown params are
# refused (400), which is why nothing unverified rides the wire.
# - MTP: the official checkpoint carries the mtp.* head (3,101 tensors,
# mtp_num_hidden_layers=1) and the plugin probes draft depth 3 — the
# publisher's throughput peak for this model. Stated explicitly with
# --num-speculative-tokens 3.
# - MTP: the .rad carries the model's mtp.* head (the publisher's recipe
# names the mtp.fc_hidden/fc_embedding/draft_head rules; card default
# depth 3). Served at the depth named by FRANK_ARCHON_SPEC_TOKENS below —
# the depth this shape's decode ladder was measured at.
# - The fused TP all-reduce is declared over max-num-seqs x
# (1 + --num-speculative-tokens) rows; 2 x 3 = 6 rows, under the 25-row
# (1 + --num-speculative-tokens) rows; 3 x (1+2) = 9 rows, under the 25-row
# exactness bound of the wht6 wire.
# Engine pin: Codeberg tag v1.0.13 (commit 53282fa), Docker Hub tag 1.0.13,
# index digest below. REJECTED UPGRADE: 1.2.0 (sha256:665775ed1d90de6dbb
@@ -79,7 +108,7 @@ FRANK_ARCHON_IMAGE=stilldeadcode/radiance@sha256:4c3f484564bb2872784e46b0a59cbd7
# .rad files; the raw HF download and its staging dir live beside it and
# are never mounted.
FRANK_ARCHON_MODEL_DIR=/home/dank/models/rad
FRANK_ARCHON_MODEL_FILE=qwen4exp-swift15-w4dn8-i8-hc8m.rad
FRANK_ARCHON_MODEL_FILE=qwen3.8-next-flash-fp8-iq4r-moe.rad
FRANK_ARCHON_STATE_DIR=/home/dank/radiance-state
# The state dir's tune/ subdirectory holds rad-tune's kernel-selection
# cache (the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune
@@ -113,18 +142,29 @@ FRANK_ARCHON_TP=2
# +5.6-6.3% prefill, ~-1.3% 8-stream decode at TP2). min-kb stays the
# engine default.
FRANK_ARCHON_TP_WIRE=wht6
# Decode slots: TWO, so the served window is a CAPACITY CONTRACT — two
# Decode slots: THREE, so the served window is a CAPACITY CONTRACT — three
# concurrent conversations of --max-model-len fit the KV pool (see the KV
# block below). The "seqs 2 cannot boot" law is NOT radiance's: it was the
# tcclaviger vLLM fork's torch.compile shape specialization on
# query_start_loc (ConstraintViolationError, measured 2026-10-02, retired
# with that runtime). Radiance booted seqs 2 at the fp8-native first boot,
# and the 2026-10-06 decode sweep measured seqs 2 and 3 statistically tied
# (30.9-33.1 t/s, inside the +-3% noise floor). Side benefit: the fused TP
# all-reduce is declared over seqs x (1 + spec) = 6 rows, under the 25-row
# bound at which the wht6 wire stays EXACT (single-sequence decode and any
# step of <= 25 rows is bit-exact; only busier steps are lossy).
FRANK_ARCHON_MAX_SEQS=2
# (30.9-33.1 t/s, inside the +-3% noise floor) — slot count buys concurrency,
# not speed, so a slot costs KV, never decode.
# Cost of each extra slot: 110 MiB of per-slot recurrent state inside the KV
# pool, and the fused TP all-reduce is declared over seqs x (1 + spec) rows —
# 3 x 3 = 9 here, under the 25-row bound at which the wht6 wire stays EXACT
# (single-sequence decode and any step of <= 25 rows is bit-exact; only busier
# steps are lossy).
# The pool is sized to this contract and nothing more (see the KV block):
# 3 x 200,000 at fp8 = 5,063 MiB, pool 5,120, and the freed 1,792 MiB went into
# the resident expert slab. The trade is explicit and it is Frank's call:
# a FOURTH full-height conversation no longer fits (5,120 MiB of pool backs
# ~608k tokens; 4 x 200,000 needs 6,601), so a fourth 200k request is refused
# as too large to fit — it is NOT squeezed, paged, or truncated. Third-party
# traffic that wants four full-height sessions is a --vram-kv-mib change, not
# a code change.
FRANK_ARCHON_MAX_SEQS=3
# Served window. Native is 262,144; 200,000 is the publisher's production
# value (compose) and what models.yml's contextWindow must quote — a longer
# prompt is refused by the engine, never truncated. The engine treats
@@ -145,25 +185,52 @@ FRANK_ARCHON_HOST_POOL_MIB=44928
# VRAM left unclaimed per card (publisher default 96 MiB).
FRANK_ARCHON_GPU_HEADROOM_MIB=96
# VRAM split of claimed memory per card, MiB (0 = derive from the card).
# THE CEILING, measured on this host 2026-10-11: card 31,856 MiB, a boot
# consumes 1,710 MiB OUTSIDE both budgets (driver context + activation arena
# 990 + hardware queues 100 + headroom 96 + step staging), and the fp8 pair
# settled at 1,379 MiB free — so `vram_weights + vram_kv <= 29,120 MiB` keeps
# >= 1 GiB free. The pair below is exactly that sum.
# THE CEILING, read off this engine's own VRAM-budget table (boot 2026-10-13,
# pair 24000/5120): card 32,624 MiB; budgets claimed 29,120; `3.42 GiB is
# neither budgeted nor accounted — the driver's context, the activation arena
# and --gpu-headroom-mib come out of it; anything past those is throughput on
# the floor`; and `card 0 settled at 2.01 GiB free against 209.19 MiB (the
# 96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the static region)`.
# So the reserve above the ENGINE's floor is 2,058 - 209 = 1,849 MiB and the
# ceiling is 29,120 + 1,849 = 30,969 MiB. The 2026-10-11 bound of 29,120
# ("31,856 - 1,710 outside both budgets - keep 1 GiB free") was MY reserve, not
# the engine's; the pair below spends the difference and lands 458 MiB above
# the engine's floor. `weights + kv <= 30,969` is the gate.
# THE PRICE, and it is paid twice: on this engine idle KV VRAM is ALSO the
# expert cache's spillover (`mover: 2275 slots can be lent out of the KV
# cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <= 256 tokens`),
# so a bigger KV pool steals weight residency from the slab AND from the
# lendable cache. Priors on this card/model: stealing 2.14 GiB of weights for
# a bigger pool (MAX_MODEL_LEN 262144) cost 2.5% decode; the EVC ladder's
# other direction (0.82 -> 0.84, +2.5 GiB weights) cost 8.9%. Expect -3..-8%
# vs the fp8 control (136 t/s median at 44.5k depth, acceptance 0.7246,
# ~190 t/s on a live 512-token probe); re-tune from a boot ledger + probe,
# never from that estimate.
FRANK_ARCHON_VRAM_WEIGHTS_MIB=22208
FRANK_ARCHON_VRAM_KV_MIB=6912
# KV cache UNQUANTIZED (bf16). This engine's flag takes ONLY bf16|fp8 — there
# is ONE kv_dtype for the whole cache, so "exact K, quantized V" (llama.cpp's
# other direction (0.82 -> 0.84, +2.5 GiB weights) cost 8.9%. The 2026-10-11
# exact-KV A/B measured the slab's own price (22208 vs the fp8-era 26071):
# prefill -29 %, TTFT at the full window +25 %, decode inside the noise band on
# 2 of 3 passes, draft acceptance UP 0.661 -> 0.740.
# 2026-10-13 (late): THREE slots, so the pool is cut to the contract (5,120)
# and every spare MiB goes to the slab: 22208 -> 24000 (the third slot's
# surplus KV), then 24000 -> 25600 (the card's settling slack). The direction
# the A/B data points: prefill is slab-limited, KV beyond the contract is not.
# The margin is thin by design (608,027 predicted tokens against the 600,000
# contract), so the boot ledger's `backs N sequences all at 200000 tokens at
# once` line is the referee for THIS pair and the concurrent probe is its
# regression test.
# VERDICT on the 25,600 arm (measured 2026-10-13, scripts/probe-frank-archon-kv.py):
# the threshold claim is FALSIFIED — the slab is a SLOPE at this end. A-cold
# 43.5k single-stream: 22,208 -> 3,397, 24,000 -> 3,290, 25,600 -> 3,584 t/s;
# the 3-way 197k arm's aggregate prefill 3,051 -> 3,589 t/s (wall
# 193.7 -> 164.5 s). So +1,600 MiB of slab = +8.9 % single-stream and
# +17.6 % concurrent, both outside the +-3 % noise floor. The 22,208/24,000
# pair landed inside noise of EACH OTHER, which is what made it read as a
# plateau; two points are not a curve.
# The cliff that remains is between 25,600 (3,584) and 26,071 (4,379, the
# 2026-10-11 fp8 control at seqs 3 / pool 3,543) — +22 % for +471 MiB. At THREE
# slots that is only reachable by breaking the ceiling (26,112 + 5,120 = 31,232
# vs 30,969, ~214 MiB above the engine's floor); at TWO slots it is free:
# pool 3,072 + slab 27,897 = exactly the ceiling, slab above the cliff.
FRANK_ARCHON_VRAM_WEIGHTS_MIB=25600
FRANK_ARCHON_VRAM_KV_MIB=5120
# KV cache QUANTIZED (fp8). This engine's flag takes ONLY bf16|fp8 — there is
# ONE kv_dtype for the whole cache, so "exact K, quantized V" (llama.cpp's
# --cache-type-k/--cache-type-v pair, which is what frank-zealot runs) is
# UNSERVABLE here: the rung below bf16 is fp8 on both halves.
# Measured per-rank cost, read out of the boot ledger's group table (fp8 boot
@@ -177,30 +244,36 @@ FRANK_ARCHON_VRAM_KV_MIB=6912
# The dtype-INDEPENDENT terms also live in the pool: kv_gdn_state 54 MiB x 2
# copies per sequence (+ conv/tail/ple ~= 110 MiB a sequence) and the prefix
# checkpoint region 8 x 55.99 = 448 MiB. So:
# pool_mib >= tokens x 13.8125 KiB + 110 x seqs + 448
# 6912 -> (6912 - 220 - 448) x 1024 / 13.8125 = 462,946 tokens; the engine
# reported 462,752 (0.04% page rounding) — 2 x 200,000 with 62k of slack,
# and the ledger says it outright: "backs 2 sequences all at 200000 tokens
# at once, which is the worst case".
# (the fp8 pool of 3543 gave 386,904 — it never held 2 x 200k, it held 1.93
# sessions, and a third long conversation preempted to the host/disk tiers).
# Secondary cost: the prefix-cache entry grows with the attention plane
# (one entry 58.50 KiB at fp8 -> 110.50 KiB at bf16, measured on the two boot
# ledgers' `kv tier transfer` lines), so the host tier below doubles to keep
# the same number of parked conversations, out of RAM the host has
# (104 GiB available with the service loaded).
FRANK_ARCHON_KV_DTYPE=bf16
# Spec decode: the checkpoint's own MTP head at the publisher's depth-3
# operating point (probe() returns 3 when mtp_num_hidden_layers > 0; the
# flag states it so the ledger records the intent).
# pool_mib >= tokens x 7.3125 KiB + 110 x seqs + 448
# 3 x 200,000 at fp8 needs 4,285 + 330 + 448 = 5,063 MiB; the pool is 5,120,
# and it predicts (5120 - 330 - 448) x 1024 / 7.3125 = 608,027 tokens: the
# three-slot contract with 1.3 % slack, every MiB above 5,063 having been
# handed to the expert slab instead. The ledger line is the referee ("backs N
# sequences all at 200000 tokens at once") — a pair this tight is read off the
# boot, never assumed.
# At bf16 the same three slots cost 8,093 + 778 = 8,871 MiB, which would cut
# the slab to 20,249 MiB (-9 %); at FOUR slots bf16 costs 10,791 + 888 =
# 11,679 MiB and the slab falls to 17,441 (-33 %). fp8 keeps 22,208 at both
# slot counts. The 2026-10-11 bf16 shape (6,912 / seqs 2) backed 462,752
# tokens = 2 x 200,000; the historical fp8 pool of 3,543 gave 386,904 (1.93
# sessions), and the four-slot fp8 boot of 2026-10-13 19:21 backed 843,276.
# Secondary saving: the prefix-cache entry shrinks with the attention plane
# (one entry 110.50 KiB at bf16 -> 58.50 KiB at fp8, measured on the two boot
# ledgers' `kv tier transfer` lines), so the host tier below parks twice as
# many conversations as it did at bf16 — no re-size needed to park three slots.
FRANK_ARCHON_KV_DTYPE=fp8
# Spec decode: the checkpoint's own MTP head, served at depth 2 — the
# publisher's card ships depth 3 as its peak; this fleet's ladder was
# measured at 2 and the swap keeps every argument, so 2 stays (rows
# 3 x (1+2) = 9, inside the wht6 25-row exactness bound).
FRANK_ARCHON_SPEC_TOKENS=2
FRANK_ARCHON_MAX_BATCHED_TOKENS=2048
# Prefix cache tiers (publisher pattern; disk tier sized for 2 slots).
# The host tier is sized in ENTRIES, and one entry doubles with the cache
# dtype (58.50 KiB at fp8 -> 117 KiB at bf16, per the ledger's `kv tier
# transfer` line): 4096 MiB at bf16 parks ~1.4 sessions of 200k where it
# parked ~2.9 at fp8. Doubled to keep the capacity, at RAM the host has
# (104 GiB available with the service loaded).
# Prefix cache tiers (publisher pattern). The host tier is sized in ENTRIES,
# and one entry tracks the cache dtype (58.50 KiB at fp8, 117 KiB at bf16, per
# the ledger's `kv tier transfer` line): at fp8, 8192 MiB parks ~5.8 sessions
# of 200k (at bf16 the same tier parked ~2.9), so four live slots plus a full
# parked set fits with the tier as it stands — at RAM the host has (104 GiB
# available with the service loaded).
FRANK_ARCHON_PREFIX_HOST_MIB=8192
FRANK_ARCHON_PREFIX_DISK_MIB=65536
# Frank's pin law: the PLE n-gram table (47.7 GiB) serves from pinned host
@@ -211,18 +284,23 @@ FRANK_ARCHON_NGRAM_PLACEMENT=ram
# prefix host = ~147 GiB) rides TTM's shared pages, capped by ttm's
# pages_limit (the current kernel exposes NO pages_max; the cap function
# takes the min over the knobs that exist). Below the plan the load dies
# mid-pin with an allocator error instead of this named refusal — raise the
# knob first (runtime sysfs write; kernel cmdline fallback + one reboot only
# if the file is read-only), see the runbook. Cap raised 2026-10-05 to
# 52,428,800 pages = 200 GiB, which still clears this floor.
# mid-pin with an allocator error instead of this named refusal.
# THE CAP IS A RUNTIME KNOB — the kernel default is half of RAM, so every
# raise dies with a reboot. Measured the hard way: the 2026-10-05 manual raise
# to 52,428,800 pages (200 GiB) was gone after the 2026-10-12 23:17 POWER
# FAILURE, the cap fell back to 125,988 MiB, and the preflight refused the
# start 6,919 times in ~20 h while Restart=always re-ran it. The gate in
# scripts/ttm-preflight.sh now RAISES the knob itself before refusing (the
# root unit is privileged; the knob is a 0644 root module parameter), so the
# recovery is a boot, not a visit to the machine.
FRANK_ARCHON_TTM_MIN_MIB=150000
FRANK_ARCHON_TTM_TARGET_MIB=204800
# Escape hatch for boot-ladder experiments. Unset -> no extra argv. Keep
# multi-flag values as one string; the serve script expands it unquoted on
# purpose.
FRANK_ARCHON_EXTRA_ARGS=
# The .rad's minimum size in MiB (deployer gate; a truncated convert is a
# guaranteed long-timeout boot instead of a fast failure). The converted
# container measured 177,109 MiB (1,857,127,997,440 B = 173.0 GiB; the
# plan-only estimate 172.76 GiB covers the data planes but not the metadata
# tail) - floor sits just under the payload.
# The .rad's minimum size in MiB (deployer gate; a truncated fetch is a
# guaranteed long-timeout boot instead of a fast failure). The deployed
# publisher container measures 116,322 MiB (121,969,901,568 B); the
# in-house w4dn8 rollback measured 116,416 MiB. Floor sits just under both.
FRANK_ARCHON_RAD_MIN_MIB=110000
+13 -6
View File
@@ -1,9 +1,9 @@
# frank-archon.service — radiance (StillDeadcode, Apache-2.0) endpoint
# (OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 at NATIVE block-FP8 precision, the
# rad-convert container qwen4exp-fp8-native.rad) on BOTH R9700s (TP=2,
# wht6 wire), dankserver :7770. 2026-10-05 cutover from FreeToken (git tag
# frank-archon-freetoken-pr587-20261005 is the pre-cutover state; the
# retired unit/env/serve trio and Dockerfile are in that tag).
# (the publisher's 4-bit-codebook container of the official
# Qwen/Qwen3.8-Flash-Next, qwen3.8-next-flash-fp8-iq4r-moe.rad, fetched by
# scripts/download-frank-archon-iq4r.sh) on BOTH R9700s (TP=2, wht6 wire),
# dankserver :7770. 2026-10-15 payload swap from the in-house w4dn8 (the
# swap commit is the revert; its .rad stays on disk).
#
# Same port, same served alias, same container identity as the unit name;
# the harness models.yml id is unchanged. The serve script drives
@@ -19,7 +19,7 @@
# unit grants LimitMEMLOCK=infinity to the dank process via PID1; the
# engine still runs out of dank's rootless podman storage.
[Unit]
Description=frank-archon — radiance Qwen3.8-Flash-Next-FP8 (native) on 2x R9700 TP=2
Description=frank-archon — radiance Qwen3.8-Flash-Next (publisher iq4r) on 2x R9700 TP=2
After=network-online.target
Wants=network-online.target
# Never let the start-rate limiter give up after a crash loop (a failed LLM
@@ -35,6 +35,13 @@ Group=dank
LimitMEMLOCK=infinity
Environment="FRANK_ARCHON_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-archon.env"
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-archon.sh
# The `+` prefix runs this as root even though the service is User=dank — the
# ttm pages cap is a root-owned knob and the pinned-RAM ledger (~147 GiB) is
# unusable below it. After a reboot the knob is back at its default (half of
# RAM), so the raise must be part of the start, not a note in a runbook: the
# 2026-10-12 power failure reset it and the endpoint refused to boot 6,919
# times. The serve script re-verifies the cap after this hook.
ExecStartPre=+/home/dank/serverAppConfigs/scripts/ttm-preflight.sh --raise
# systemd stop: wait alone cannot stop the container child; stop it
# explicitly. No graceful drain contract is claimed (the OpenAI front end
# holds no queue to drain — in-flight decodes are lost, which a stop
+53 -43
View File
@@ -1,59 +1,69 @@
# frank-zealot — llama.cpp (master, CUDA sm_86) serving
# HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090,
# :7771.
# frank-zealot — llama.cpp (PrismML fork build, CUDA) serving
# prism-ml/Ternary-Bonsai-2-27B-gguf at PQ2_0 on the RTX 3090, :7771.
#
# 2026-10-07 (Frank): swap from Gemma4-26B-A4B QAT Q4_K_M (one 200,704 slot)
# to Qwen3.5-9B Q6_K at TWO 200,704-token conversations, EXACT (f16) KV,
# MTP disabled (SPEC_TYPE=none), log verbosity 4.
# 2026-10-17 (Frank): swap from Qwen3.5-9B-Uncensored Q6_K to the Bonsai-2-27B
# ternary checkpoint at the SAME shape Frank asked for — TWO 200,704-token
# conversations — with q8_0 K + q8_0 V (f16 is arithmetically impossible here,
# see the KV law), MTP off (the file carries no nextn/mtp tensors), verbosity 4.
#
# Architecture — read from the GGUF's own header (HTTP range read, not the
# model card): arch qwen35, 32 layers, dense 9B, full_attention_interval 4
# ⇒ 8 full-attention layers (3,7,11,...,31) + 24 Gated DeltaNet linear layers,
# 16 attention heads / 4 KV heads × 256 dim, ssm conv_kernel 4 / state_size
# 128 / group_count 16 / time_step_rank 32 / inner_size 4096, native 262,144,
# vocab 248,320, file_type 18 (Q6_K). Tensor-type histogram: Q6_K×250 +
# F32×177 — standard types only, so the stock master build loads it (no
# custom-quant lock-in like the ROCmFPX era). NO nextn/mtp tensors exist, so
# in-file MTP is impossible here; SPEC_TYPE=none is forced by the payload.
# THE RUNTIME IS A LOCK, NOT A PREFERENCE. The file is general.file_type 141
# with 402 tensors of the CUSTOM GGML type 142 (pq2_0) plus prism.hadamard.*
# metadata (normalized-Sylvester-Walsh transform, block 1024, 401 named
# weights, inverse on token_embd, gdn_v_grouped). Stock llama.cpp refuses type
# 142 at load; only the PrismML fork registers the type AND reads the Hadamard
# metadata. The vendor prebuilt build 10754 (commit 2459f68b5) is what every
# number in this file was measured on; sha256 12ae5baba033d8cd…
LLAMA_BIN=/home/dank/GitPackages/llama-prism/llama-server
MODEL_FILE=/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-PQ2_0.gguf
MMPROJ_FILE=/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf
# The CRACK re-export (dealignai/Bonsai-2-27B-Ternary-CRACK-GGUF, uncensored,
# already on disk at /home/dank/models/Bonsai2-CRACK/) is a ONE-LINE swap: the
# header census proves byte-identical geometry (851 tensors, same
# {pq2_0:402, f32:353, bf16:96} histogram, 64 blocks, vocab 248,320, same
# 8,952-char chat template) and the official file's sha256 is the Hub LFS oid.
# Bonsai-2 is the BASE (censored) Qwen3.8 family; if a compaction or subagent
# request ever comes back with a refusal, point MODEL_FILE at the CRACK file
# and restart — the VRAM ledger below does not move.
#
# KV rate law: 8 full layers × 4 KV heads × 256 dim = 8,192 elements per
# token per cache. f16 K + f16 V = 32,768 B/token = 32 KiB/token, so the
# 401,408-token pool is 12.25 GiB. The 24 DeltaNet layers cost a fixed
# recurrent state per SLOT (not per token) — that is why two 200k slots fit
# where the 26B MoE fit only one.
#
# Feasibility comes from the predecessor's MEASURED boots, not a formula:
# MiMo-V2.6-Distill-Qwen-9B (identical qwen35 geometry, 7,793,710,624 B) at
# 2×200,704 f16 K + f16 V measured 21,276 used / 2,849 free at ubatch 1024
# and 21,846 / 2,279 at ubatch 2048. This file is 415 MiB lighter in weights,
# its BF16 mmproj is 3 MiB heavier ⇒ projected ~21,430 used / ~2,700 free.
# The verbosity-4 boot ledger is the referee.
#
# Ubatch 2048 is the measured CUDA optimum (measured on the MiMo predecessor
# at this exact arch + shape: +4.6 % prefill at 8.9k depth, decode flat, cost
# 616 MiB). 1024 is the fallback rung if the boot lands under the 2 GiB floor.
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
MODEL_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf
MMPROJ_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf
# Served model id — the harness/family name (Protoss rename completed to the
# wire 2026-10-09: harness ids + homepage cards 2026-10-08, /v1/models aliases
# this day). The SERVICE, unit, scripts and this env file keep the
# frank-zealot name; only the alias on the wire changed. Consumers that
# key on the served id: OpenWebUI (model row, access_grant, api_configs
# model_ids) and the harness models.yml.
# Served model id — the harness/family name. The SERVICE, unit, scripts and
# this env file keep the frank-zealot name; only the alias on the wire is the
# model id. Consumers that key on the served id: OpenWebUI (model row,
# access_grant, api_configs model_ids) and the harness models.yml.
ALIAS=frank-zealot
BIND_ADDR=0.0.0.0
LISTEN_PORT=7771
# CTX_SIZE is the POOL llama.cpp divides across slots: 401,408 / 2 = 200,704
# per conversation. MUST stay divisible by PARALLEL (the seam enforces it) and
# the harness contextWindow quotes the PER-SLOT number — a longer prompt is
# answered with a 400, never truncated.
# answered with a 400, never truncated. The model's native window is 262,144;
# 2 × 262,144 at q8_0 projects 25,758 MiB and does NOT fit this card (measured
# f16 analogue: 41,118). One 262,144 slot fits with 5,995 MiB free but
# serializes compaction, so the pair stays.
CTX_SIZE=401408
PARALLEL=2
BATCH_SIZE=4096
UBATCH_SIZE=2048
CACHE_TYPE_K=f16
CACHE_TYPE_V=f16
# KV law (from the GGUF header, then confirmed by the boot ledger): 16
# full-attention layers (full_attention_interval 4 over 64 blocks) × 4 KV heads
# × 256 dim = 16,384 elements per token per cache. q8_0 bills 17,408 B per
# cache ⇒ 34,816 B/token = 34 KiB, so the 401,408 pool is 13,328.00 MiB — the
# ledger said exactly that. f16 would bill 25,088 MiB and the boot DIES:
# `cudaMalloc failed: out of memory` on the 25,088.00 MiB request, engine fit
# projection 33,198 vs 23,558 free. The 48 DeltaNet layers bill a fixed
# recurrent state per SLOT — ~150 MiB each (context 13,627 − KV 13,328 = 299
# for two slots), which is why pool arithmetic alone under-prices the shape.
CACHE_TYPE_K=q8_0
CACHE_TYPE_V=q8_0
# Boot ledger (2026-10-17, this exact shape, nvidia-smi settled):
# 22,664 MiB used / 1,461 MiB free of 24,123
# fit projection 21,439 = 6,539 model + 13,627 context + 1,272 compute
# the mtmd projector's 873.10 MiB worst-case is NOT inside that projection —
# it lands after the fit check, which is the whole difference between
# "2.6 GiB free on paper" and 1.4 GiB free in reality. Watch it.
# Fallback rung if the margin is ever eaten (desktop, co-tenant): q4_0 KV
# measured 16,392 used / 7,733 free at the same 2 × 200,704.
# 1024 is the ubatch fallback (−616 MiB, measured on the 9B predecessor at
# this card/arch; 2048 was the measured CUDA optimum, +4.6 % prefill).
SPEC_TYPE=none
NGL=99
# Log verbosity threshold (llama.cpp -lv). 4 = debug: KV layout, buffer sizes
+1 -1
View File
@@ -1,5 +1,5 @@
[Unit]
Description=llama.cpp Qwen3.5-9B-Uncensored-HauhauCS-Aggressive Q6_K GGUF (qwen35 hybrid, 2x200704 slots, f16 K+V, MTP off, mmproj vision) on the RTX 3090 (dankclaw, frank-zealot)
Description=llama.cpp (PrismML build 10754) Ternary-Bonsai-2-27B PQ2_0 GGUF (qwen35 hybrid, 2x200704 slots, q8_0 K+V, MTP off, mmproj vision) on the RTX 3090 (dankclaw, frank-zealot)
After=network-online.target
Wants=network-online.target
+66 -12
View File
@@ -1,22 +1,30 @@
#!/usr/bin/env python3
"""probe-frank-archon-kv.py — verify the 2 x 200,000 KV contract on a LIVE
"""probe-frank-archon-kv.py — verify the seqs x max-model-len KV contract on a
frank-archon (radiance) service. Run it ON dankserver: it talks to
127.0.0.1:7770 and builds its prompts from the repo checkout there.
No scratch container, no mask: the service under test is the service.
ssh dankserver 'python3 /home/dank/serverAppConfigs/scripts/probe-frank-archon-kv.py'
Three probes, in increasing cost:
Four probes, in increasing cost:
A decode at depth — one ~51k-token chat completion, 512 tokens, greedy.
Reports the engine's own `timings` (prefill, decode, draft acceptance).
The bench law applies: >=128 generated tokens, greedy, real text.
B the capacity contract — TWO concurrent chats of ~197k prompt tokens
(~394k tokens resident) each carrying a unique marker at its head. Both
must answer their own marker. This is what a 2 x 200,000 endpoint owes:
two full-height conversations resident at once, exact retrieval in both.
B the capacity contract — as many CONCURRENT chats of ~197k prompt tokens as
the deployed `FRANK_ARCHON_MAX_SEQS` (read from the repo env file on this
host), each carrying a unique marker at its head. All must answer their own
marker. This is what the endpoint owes: N full-height conversations
resident at once, exact retrieval in every one of them.
C prefix-cache restore — probe A's cold prompt again, so the disk tier has
to give the blocks back (cache_n > 0, TTFT collapses).
D concurrency decode — `seqs` CONCURRENT 512-token generations at ~25 % of
the window. This is the cost a slot count imposes on DECODE, which no
prefill number shows: the fused-step schedule is unfair, so every stream is
printed plus the aggregate. Measured 2026-10-13 at 3 x 200,000 fp8:
single-stream 99.0 t/s (43.5k) and 93.3 (197k) — decode is nearly
window-free — against 3-stream 113.9/43.3/10.9 t/s = 18.4 t/s aggregate.
Concurrency, not the slab, is what decode pays for slots. The 2026-10-13
shape hunt missed this for a whole day because leg B generates 24 tokens.
PROBE LAW (learned the hard way, twice): radiance routes the thinking block to
`reasoning_content`, so a chat request whose budget is spent inside
`<think>` returns EMPTY `content` with `finish_reason: stop`. A probe that
@@ -114,23 +122,51 @@ def show(tag, j, wall, expect=None):
print(line)
def deployed_shape():
"""The contract is the deployed env, not a constant: slot count and window
come from the checkout on this host, so the probe cannot drift from the
service it is testing (the 2026-10-13 shape is 4 x 200,000; before that
2 x 200,000, and the fp8 era 3 x 200,000)."""
shape = {}
with open(f"{REPO}/scripts/frank-archon.env") as fh:
for line in fh:
line = line.strip()
if line.startswith("FRANK_ARCHON_") and "=" in line:
key, _, val = line.partition("=")
shape[key] = val.strip().strip('"')
return (int(shape["FRANK_ARCHON_MAX_SEQS"]),
int(shape["FRANK_ARCHON_MAX_MODEL_LEN"]),
shape["FRANK_ARCHON_KV_DTYPE"])
def main():
seqs, maxlen, dtype = deployed_shape()
print(f"shape: {seqs} x {maxlen:,} tokens, kv {dtype}")
print("health:", json.dumps(get("/health")))
print("models:", [m.get("id") for m in get("/v1/models").get("data", [])])
deep = corpus(160_000)
wide = corpus(1_100_000)
facts = {"B1": "cobalt-harbor-13", "B2": "quartz-meadow-42"}
bodies = {s: chat(wide, 646_000, s, f, 24) for s, f in facts.items()}
# 3.28 chars/token is the measured ratio of this corpus through the
# froggeric template (646,000 chars -> 196,919 tokens at the 2026-10-11
# deploy); 98.5 % of the window leaves the response room under the ceiling.
chars = int(0.985 * maxlen * 3.28)
# Each marker's FIRST hyphen-separated word is unique, because show()
# scores the hit on that word — a shared prefix would score every slot HIT
# on any answer that says the word.
words = ["cobalt", "quartz", "basalt", "ember", "willow", "cobweb", "flint"]
facts = {f"B{i+1}": f"{words[i % len(words)]}-flag-{(i + 1) * 137}"
for i in range(seqs)}
bodies = {s: chat(wide, chars, s, f, 24) for s, f in facts.items()}
print(f"[B] {len(bodies['B1']['messages'][0]['content']):,} chars per "
f"request, x2 concurrent")
f"request, x{seqs} concurrent")
# A + C first (cheap), B last (the expensive pair).
# A + C first (cheap), B last (the expensive set).
for tag, salt in (("A-cold", "A1-9137"), ):
j, wall, _ = post(chat(deep, 140_000, salt, "zephyr-lantern-7", 512))
show(tag, j, wall, "zephyr-lantern-7")
t0 = time.time()
with concurrent.futures.ThreadPoolExecutor(max_workers=2) as ex:
with concurrent.futures.ThreadPoolExecutor(max_workers=seqs) as ex:
futs = {s: ex.submit(post, b) for s, b in bodies.items()}
for s, fut in futs.items():
try:
@@ -139,6 +175,24 @@ def main():
except Exception as e: # a preemption/timeout lands here
print(f"[B-{s}] FAILED after {time.time() - t0:.1f}s: {e!r}")
print(f"[B] total {time.time() - t0:.1f}s")
# D — concurrency decode: seqs streams, 512 tokens each, ~25 % depth.
dchars = int(0.25 * maxlen * 3.28)
dbodies = [chat(deep, dchars, f"D{i + 1}-9137", f"flint-marker-{i}", 512)
for i in range(seqs)]
print(f"[D] {seqs} concurrent streams x 512 tokens, "
f"{dchars:,} chars each")
t0 = time.time()
with concurrent.futures.ThreadPoolExecutor(max_workers=seqs) as ex:
futs = [ex.submit(post, b) for b in dbodies]
for i, fut in enumerate(futs, 1):
try:
j, wall, _ = fut.result()
show(f"D-s{i}", j, wall)
except Exception as e: # a preemption/timeout lands here
print(f"[D-s{i}] FAILED after {time.time() - t0:.1f}s: {e!r}")
tot = time.time() - t0
print(f"[D] {seqs} x 512 tokens in {tot:.1f}s = "
f"{seqs * 512 / tot:.1f} t/s aggregate decode")
j, wall, _ = post(chat(deep, 140_000, "A1-9137", "zephyr-lantern-7", 512))
show("C-prefix-hit", j, wall, "zephyr-lantern-7")
+27 -37
View File
@@ -1,11 +1,13 @@
#!/usr/bin/env bash
# serve-frank-archon.sh — frank-archon endpoint: radiance (StillDeadcode,
# Apache-2.0) serving the OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint at
# NATIVE block-FP8 precision from a rad-convert container
# (/home/dank/models/rad/qwen4exp-fp8-native.rad), on BOTH R9700s (TP=2,
# wht6 wire), dankserver :7770. 2026-10-05 cutover from FreeToken (PR #587,
# one card; git tag frank-archon-freetoken-pr587-20261005 is the
# pre-cutover state).
# Apache-2.0) serving the PUBLISHER's 4-bit-codebook container of the
# official Qwen/Qwen3.8-Flash-Next (the engine author's own rad-convert of
# the bf16 checkpoint: qwen3.8-next-flash-fp8-iq4r-moe.rad, fetched by
# scripts/download-frank-archon-iq4r.sh) on BOTH R9700s (TP=2, wht6 wire),
# dankserver :7770. 2026-10-15 payload swap from the in-house w4dn8 of the
# Swift1.5 finetune (payload on disk = rollback; the swap commit is the
# revert); 2026-10-05 radiance cutover from FreeToken (git tag
# frank-archon-freetoken-pr587-20261005 is the pre-radiance state).
#
# The run line below IS the engine argv: the image's entrypoint is the
# engine and its tools sit beside it on PATH. Unlike every predecessor
@@ -13,9 +15,11 @@
# log states every budget before anything is allocated (weights vs KV
# split, the expert host pool's coverage of the worst case, the PLE pin,
# the P2P answer per pair) — read it after every resize.
# Deployed shape (2026-10-11): 2 decode slots x 200,000 tokens, bf16
# (unquantized) KV in a 6,912 MiB pool, the weights slab cut to 22,208 MiB to
# pay for it, 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache host
# Deployed shape (2026-10-13): THREE decode slots x 200,000 tokens, fp8
# (quantized) KV in a 5,120 MiB pool (contract 3 x 200,000 + 1.3 %) and a
# 25,600 MiB resident expert slab — 30,720 MiB of the 30,969 MiB the engine's
# own budget table makes claimable, spent on the slab because prefill is
# slab-limited. 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache host
# 8 GiB + disk 64 GiB, PLE pinned in RAM.
#
# The root unit (scripts/frank-archon.service, User=dank,
@@ -61,6 +65,7 @@ set +a
: "${FRANK_ARCHON_PREFIX_DISK_MIB:?FRANK_ARCHON_PREFIX_DISK_MIB must be set}"
: "${FRANK_ARCHON_NGRAM_PLACEMENT:?FRANK_ARCHON_NGRAM_PLACEMENT must be set}"
: "${FRANK_ARCHON_TTM_MIN_MIB:?FRANK_ARCHON_TTM_MIN_MIB must be set}"
: "${FRANK_ARCHON_TTM_TARGET_MIB:?FRANK_ARCHON_TTM_TARGET_MIB must be set}"
: "${FRANK_ARCHON_RAD_MIN_MIB:?FRANK_ARCHON_RAD_MIN_MIB must be set}"
# The image gate is the digest-pinned ref itself: a name@sha256 ref exists
@@ -117,34 +122,19 @@ if [ ! -e /dev/kfd ] || [ ! -d /dev/dri ]; then
exit 1
fi
# TTM preflight: the pinned ledger (~2 x 40 GiB pools + 47.7 GiB PLE + 4 GiB
# prefix host = ~132 GiB) rides TTM's shared pages, capped by ttm's
# pages_limit (the current kernel exposes NO pages_max; the cap function
# takes the min over the knobs that exist). Below FRANK_ARCHON_TTM_MIN_MIB
# the load dies mid-pin with an allocator error instead of this named
# refusal — raise the knob first (runtime sysfs write; kernel cmdline
# fallback + one reboot only if the file is read-only), see the runbook.
# RAISED 2026-10-05 pre-cutover to 52,428,800 pages = 200 GiB.
ttm_cap_mib() {
# min over the knobs THIS kernel exposes — the current kernel has
# pages_limit only (no pages_max); a knob that is absent is skipped,
# not a zero (a zero would refuse a healthy cap).
local v cur=""
for p in /sys/module/ttm/parameters/pages_limit /sys/module/ttm/parameters/pages_max; do
[ -r "$p" ] || continue
v="$(cat "$p" 2>/dev/null)" || continue
[ -n "$v" ] || continue
v=$(( v * 4 / 1024 ))
if [ "$v" -lt "${cur:-999999999}" ]; then cur=$v; fi
done
echo "${cur:-0}"
}
TTM_CAP="$(ttm_cap_mib)"
if [ "$TTM_CAP" -lt "$FRANK_ARCHON_TTM_MIN_MIB" ]; then
echo "ERROR: ttm pages cap is ${TTM_CAP} MiB, the pinned ledger needs >= $FRANK_ARCHON_TTM_MIN_MIB" >&2
echo "frank-archon: raise /sys/module/ttm/parameters/pages_limit (and pages_max if the kernel exposes one) to >= $FRANK_ARCHON_TTM_MIN_MIB in 4 KiB pages" >&2
exit 1
fi
# TTM preflight: the pinned ledger (2 x FRANK_ARCHON_HOST_POOL_MIB + the
# 47.7 GiB PLE table + the prefix host tier, ~147 GiB) rides TTM's shared
# pages, capped by ttm's pages_limit (the current kernel exposes NO
# pages_max; the cap function takes the min over the knobs that exist).
# Below FRANK_ARCHON_TTM_MIN_MIB the load dies mid-pin with an allocator
# error, so the gate refuses first — and RAISES the knob before refusing:
# the cap is a runtime knob whose default is half of RAM, so a power failure
# resets it (measured 2026-10-12 23:17 on this host: 200 GiB -> 125,988 MiB,
# and Restart=always re-ran the refusal 6,919 times over ~20 h).
# FRANK_ARCHON_TTM_TARGET_MIB is the recorded host ceiling, not the floor.
# shellcheck source=/dev/null
source "$SCRIPT_DIR/ttm-preflight.sh"
ttm_ensure_cap || exit 1
# A crashed/killed unit can leave the container name registered (systemd
# SIGKILLs conmon before cleanup runs). Clear OUR name plus the legacy
+38 -35
View File
@@ -1,28 +1,30 @@
#!/usr/bin/env bash
# Serve HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive (Q6_K) as
# frank-zealot on Dankclaw's RTX 3090 via llama.cpp (master build, CUDA sm_86).
# Serve prism-ml/Ternary-Bonsai-2-27B (PQ2_0 ternary quant) as frank-zealot on
# Dankclaw's RTX 3090 via llama.cpp (PrismML vendor build 10754, CUDA sm_86).
#
# 2026-10-07 (Frank): Qwen3.5-9B swap at TWO 200,704-token conversations
# (--ctx-size 401408 --parallel 2) with EXACT f16 KV — the 9B rung frees the
# room the 26B MoE Gemma4 shape lacked, so quantized V retires. MTP stays off:
# this GGUF carries no MTP tensors at all (verified from its tensor-name
# list), so there is no in-file head to draft with. Verbosity 4 ships so the
# VRAM-fit forensics (KV layout, buffer sizes, graph capture) land in the journal.
# 2026-10-17 (Frank): Bonsai-2 PQ2_0 swap at TWO 200,704-token conversations
# (--ctx-size 401408 --parallel 2) with q8_0 K + q8_0 V — f16 KV does not fit
# this geometry: the engine's own fit projected 33,198 MiB against 23,558 free
# and cudaMalloc failed on the 25,088 MiB pool. MTP stays off: this GGUF
# carries no MTP tensors (verified from its tensor-name list). Verbosity 4
# ships so the VRAM-fit forensics (KV layout, buffer sizes, graph capture)
# land in the journal.
#
# Architecture: qwen35 — dense 9B, 32 layers with full_attention_interval 4
# (8 full-attention layers at 3,7,...,31 + 24 Gated DeltaNet linear layers),
# 16 heads / 4 KV heads × 256 dim, native 262,144, vocab 248,320. The KV pool
# scales with the 8 full layers: 32 KiB/token at f16 K + f16 V (12.25 GiB at
# the 401,408 pool); the DeltaNet layers cost a fixed recurrent state per
# slot. Feasibility is priced from the MiMo-V2.6-Distill-Qwen-9B boots (same
# geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279
# at ubatch 2048 — the boot ledger, not arithmetic, is the referee.
# Vision comes from the publisher's own BF16 projector (the model is natively
# multimodal), so --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is
# the documented text-only opt-out.
# Architecture: qwen35 — 27B, 64 blocks with full_attention_interval 4 (16
# full-attention layers + 48 Gated DeltaNet linear layers), 24 heads / 4 KV
# heads × 256 dim, native 262,144, vocab 248,320. The KV pool scales with the
# 16 full layers: 34 KiB/token at q8_0 K + q8_0 V (13,328 MiB at the 401,408
# pool); the DeltaNet layers cost a fixed recurrent state per SLOT (~150 MiB —
# boot-ledger context minus KV). Boot ledger: 22,664 used / 1,461 free, with
# the Q8_0 projector's 873 MiB landing AFTER the engine's fit projection.
# The runtime is a hard lock, not a preference: the file carries 402 tensors
# of custom GGML type 142 plus prism.hadamard.* metadata, which only the
# vendor build registers — stock llama.cpp refuses it at load.
# Vision comes from the publisher's Q8_0 projector, so --mmproj is what gives
# this endpoint eyes; MMPROJ_FILE="" is the documented text-only opt-out.
#
# Sampling: llama.cpp takes it per request (the fleet harness sends the
# publisher recipe from models.yml: thinking temp 0.6 / top_p 0.95 / top_k 20).
# Sampling: llama.cpp takes it per request (the fleet harness sends the file's
# own template defaults from models.yml: thinking temp 1.0 / top_p 0.95 / top_k 20).
#
# Tunables come from frank-zealot.env unless FRANK_ZEALOT_ENV_FILE overrides.
set -euo pipefail
@@ -37,8 +39,8 @@ if [[ -f "$ENV_FILE" ]]; then
fi
# --- defaults (used when the env file does not set them) -------------------
: "${LLAMA_BIN:=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
: "${MODEL_FILE:=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf}"
: "${LLAMA_BIN:=/home/dank/GitPackages/llama-prism/llama-server}"
: "${MODEL_FILE:=/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-PQ2_0.gguf}"
: "${ALIAS:=frank-zealot}"
: "${BIND_ADDR:=0.0.0.0}"
: "${LISTEN_PORT:=7771}"
@@ -54,19 +56,20 @@ fi
# throughput-neutral at a fixed ubatch.
: "${BATCH_SIZE:=4096}"
: "${UBATCH_SIZE:=2048}"
# Exact (unquantized) KV: f16 K + f16 V, restoring the MiMo-era contract that
# the 26B Gemma4 shape could not afford. 32 KiB/token at this geometry
# (8 full-attention layers × 4 KV heads × 256 dim); 12.25 GiB at the 401,408
# pool. Explicit either way so the contract is visible in the argv, and
# --flash-attn stays required for any quantized-V fallback (q8_0 = llama.cpp's
# FP8 class; no true fp8 KV dtype exists in this build).
: "${CACHE_TYPE_K:=f16}"
: "${CACHE_TYPE_V:=f16}"
# Speculative decoding is OFF: this Q6_K file carries NO nextn/mtp tensors, so
# Quantized KV: q8_0 K + q8_0 V — the only pair that funds 2 × 200,704 on this
# card. 34 KiB/token at this geometry (16 full-attention layers × 4 KV heads ×
# 256 dim); 13,328 MiB at the 401,408 pool. f16 measured OOM; q4_0 is the
# fallback rung (16,392 used / 7,733 free). Explicit either way so the contract
# is visible in the argv, and --flash-attn stays required for quantized V
# (no true fp8 KV dtype exists in this build).
: "${CACHE_TYPE_K:=q8_0}"
: "${CACHE_TYPE_V:=q8_0}"
# Speculative decoding is OFF: this PQ2_0 file carries NO nextn/mtp tensors, so
# there is no in-file head (the law: MTP is a property of the file, not of the
# flags). SPEC_TYPE=none strips BOTH speculative flags (flags move as a set);
# naming a drafter requires SPEC_DRAFT_FILE alongside SPEC_TYPE — the Gemma4
# assistant on disk is arch gemma4-assistant and cannot drafter-match qwen35.
# an MTP-bearing CRACK re-export exists upstream (ManniX-ITA, +451 MB) if
# decode ever needs it — that is a new payload with its own VRAM bill, never a
# flag flip on this file.
: "${SPEC_TYPE:=none}"
: "${SPEC_DRAFT_N_MAX:=3}"
: "${SPEC_DRAFT_FILE:=}"
@@ -77,7 +80,7 @@ fi
: "${VERBOSITY:=3}"
# Vision projector. No colon on purpose: an explicitly empty value means
# "text-only, on purpose" and must NOT be replaced by the default path.
: "${MMPROJ_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf}"
: "${MMPROJ_FILE=/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf}"
# Fail loudly rather than half-loading: a missing file is a download that
# has not finished, and Restart=always would turn it into a crash loop that
+104
View File
@@ -0,0 +1,104 @@
#!/usr/bin/env bash
# ttm-preflight.sh — the TTM pages-cap gate for engines that PIN host RAM.
#
# Sourced, never executed. An engine that pins tens of GiB of host RAM (the
# radiance expert pools + the PLE n-gram table) has its pinned pages charged
# against the kernel's ttm page cap: /sys/module/ttm/parameters/pages_limit
# (and pages_max on kernels that expose one). The cap function below takes the
# MIN over the knobs the running kernel actually exposes — an absent knob is
# skipped, never read as zero, because a zero would refuse a healthy cap.
#
# THE LAW THIS FILE EXISTS FOR: the cap is a RUNTIME knob. The kernel default is
# half of RAM, so every raise is lost on reboot. On 2026-10-12 23:17 a power
# failure on dankserver reset it 200 GiB -> 125,988 MiB, and frank-archon's
# preflight refused the start — 6,919 crash-restarts over ~20 hours, with
# `Restart=always` cheerfully re-running a refusal. A manual sysfs write is
# therefore not an acceptable repair procedure: the gate raises the knob itself
# (the units that need it run as root, and the knob is a 0644 root-owned module
# parameter), re-reads, and only then refuses.
#
# Contract (set by the sourcing script's env file):
# FRANK_ARCHON_TTM_MIN_MIB floor the pinned ledger needs; refuse below it
# FRANK_ARCHON_TTM_TARGET_MIB value to raise the cap to when it is below the
# floor (the recorded host ceiling — writing the
# floor itself would leave the ledger no slack)
# FRANK_ARCHON_TTM_SYS_DIR knob directory; defaults to the real sysfs
# path, a test points it at a temp dir.
# Pages are 4 KiB: pages = MiB x 256.
: "${FRANK_ARCHON_TTM_SYS_DIR:=/sys/module/ttm/parameters}"
# Prints the effective cap in MiB, 0 when no knob is readable.
ttm_cap_mib() {
local p v cur=""
for p in "$FRANK_ARCHON_TTM_SYS_DIR/pages_limit" \
"$FRANK_ARCHON_TTM_SYS_DIR/pages_max"; do
[ -r "$p" ] || continue
v="$(cat "$p" 2>/dev/null)" || continue
[ -n "$v" ] || continue
v=$(( v * 4 / 1024 ))
if [ "$v" -lt "${cur:-999999999}" ]; then cur=$v; fi
done
echo "${cur:-0}"
}
# Writes $1 MiB (as 4 KiB pages) into every knob present. Failures are
# swallowed: the caller re-reads the cap, and a write the kernel refused shows
# up there (ttm refuses a limit below the pages it has already pinned).
#
# PRIVILEGE, measured the hard way: the knob is root-owned 0644 and the service
# process runs as User=dank, so `-w` is FALSE there and a plain echo fails. The
# canonical privileged path is the unit's ExecStartPre with the `+` prefix
# (systemd runs it as root even under User=); `sudo -n` is the fallback for a
# human running the gate from a shell (passwordless sudo on both hosts).
ttm_raise_cap() {
local pages=$(( $1 * 256 )) p
for p in "$FRANK_ARCHON_TTM_SYS_DIR/pages_limit" \
"$FRANK_ARCHON_TTM_SYS_DIR/pages_max"; do
[ -e "$p" ] || continue
if [ -w "$p" ]; then
echo "$pages" > "$p" 2>/dev/null || true
elif command -v sudo >/dev/null 2>&1; then
sudo -n sh -c "echo $pages > '$p'" >/dev/null 2>&1 || true
fi
done
}
# Returns 0 when the cap clears FRANK_ARCHON_TTM_MIN_MIB, raising it first if
# needed; returns 1 with a named refusal when it cannot be made to.
ttm_ensure_cap() {
local cap
cap="$(ttm_cap_mib)"
if [ "$cap" -lt "$FRANK_ARCHON_TTM_MIN_MIB" ]; then
echo "frank-archon: ttm pages cap is ${cap} MiB, below the pinned-ledger" \
"floor $FRANK_ARCHON_TTM_MIN_MIB — raising the knob to" \
"$FRANK_ARCHON_TTM_TARGET_MIB MiB ($(( FRANK_ARCHON_TTM_TARGET_MIB * 256 )) pages)" >&2
ttm_raise_cap "$FRANK_ARCHON_TTM_TARGET_MIB"
cap="$(ttm_cap_mib)"
fi
if [ "$cap" -lt "$FRANK_ARCHON_TTM_MIN_MIB" ]; then
echo "ERROR: ttm pages cap is ${cap} MiB, the pinned ledger needs >= $FRANK_ARCHON_TTM_MIN_MIB" >&2
echo "frank-archon: raise $FRANK_ARCHON_TTM_SYS_DIR/pages_limit (and pages_max if the kernel exposes one) to >= $(( FRANK_ARCHON_TTM_MIN_MIB * 256 )) in 4 KiB pages; a read-only knob needs the module option (options ttm pages_limit=N) and a reboot" >&2
return 1
fi
echo "frank-archon: ttm pages cap ${cap} MiB clears the pinned-ledger floor $FRANK_ARCHON_TTM_MIN_MIB" >&2
}
# Executed (not sourced) it is the privileged pre-start hook: systemd runs this
# as root from the unit's `ExecStartPre=+…`, where the knob write is legal and
# the `Restart=always` loop can never outrun it.
if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
set -euo pipefail
case "${1:-}" in
--raise)
TTM_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=/dev/null
source "${FRANK_ARCHON_ENV_FILE:-$TTM_DIR/frank-archon.env}"
ttm_ensure_cap
;;
*)
echo "usage: ${0##*/} --raise (raise the ttm pages cap, then verify)" >&2
exit 2
;;
esac
fi
+4 -2
View File
@@ -43,8 +43,10 @@ As of 2026-10-07 the picker exposes **exactly one model: `frank-zealot`**.
| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | llama.cpp CUDA (RTX 3090, 2×200,704) | Dankclaw |
The `model` table carries exactly one decoration row, `frank-zealot`
(context length 200,704 = the per-slot window; description: Qwen3.5-9B
Q6_K + BF16 vision mmproj).
(context length 200,704 = the per-slot window; description since 2026-10-17:
Ternary-Bonsai-2-27B PQ2_0 + Q8_0 vision mmproj — the row's `description` is
live DB state, so a payload swap must re-key it in the database too, not only
in the repo).
A third gate sits *after* the table, and it is the one that bites family
accounts (2026-10-07): a model row is only visible to a non-admin user when
@@ -452,7 +452,8 @@ access is decided entirely by GROUP MEMBERSHIP consumed downstream —
- `Non-Admin Users` = the plain-user marker group (precedents: qa-e2e, and
every human account).
- **The human-user template** (ridley, kassie, alex, artem, kirby, ry, nic,
debo, freya — kirby/ry/nic/debo/freya (pk 17–21) created 2026-10-09, kassie
debo, freya, mitch, eric — mitch (pk 22) and eric (pk 23) created
2026-10-15, kirby/ry/nic/debo/freya (pk 17–21) created 2026-10-09, kassie
and alex created 2026-10-08, artem (pk 16) created 2026-10-07): `username` +
`<user>@franksdank.cloud` + display name +
`is_active=True` + the password set via `set_password`, membership in
@@ -0,0 +1,154 @@
# frank-archon — Radiance (StillDeadcode), publisher Qwen3.8-Flash-Next iq4r, 2× R9700 TP=2
**Status: LIVE — deployed 2026-10-15.** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 2× Radeon AI PRO R9700 (gfx1201), TP=2
The in-house w4dn8 conversion of the Swift1.5 finetune
([[frank-archon-radiance-swift15-w4dn8]]) is replaced by the **engine
author's published container**:
[`StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe`](https://huggingface.co/StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe)
— the official `Qwen/Qwen3.8-Flash-Next` bf16 checkpoint rad-converted by
StillDeadcode. Same port, alias, container name, systemd unit, digest-pinned
engine pin **1.0.13** (`stilldeadcode/radiance@sha256:4c3f484564…`), and
**every serving argument carried verbatim** from the 2026-10-13 shape.
Rollback: re-download the w4dn8 `.rad` from the HF mirror + point
`MODEL_FILE` at it (the local copy was purged 2026-10-17, see Rollback).
## The payload
`qwen3.8-next-flash-fp8-iq4r-moe.rad` — **121,969,901,568 B = 116,322 MiB**,
sha256 `0af5e96244e80c21ac8edca1719dae49b8a201b94a2f3994c3e24026ceaa4d20`
(Hub LFS oid), fetched byte-exact by `scripts/download-frank-archon-iq4r.sh`
(hfget resume + size + sha256; staging dir
`/home/dank/models/StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe/` keeps the
card, LICENSE, and recipe `qwen4exp-w4nl64-i8-hc8m` as provenance).
| Plane | Encoding (publisher recipe `qwen4exp-w4nl64-i8-hc8m`) |
| --- | --- |
| Routed experts | 4-bit codes into a 16-level **non-uniform** codebook after a 128-point Walsh–Hadamard rotation, one E4M3 scale per 64 weights; codes by GPTQ (10M-token calibration, per-expert down-proj Hessians) + 3 coordinate-descent sweeps |
| Protected experts | **ten** per layer, whole-expert bf16 (down-projection energy) |
| Trunk + lm_head | int8 W8A8, scale per 128 columns searched; HC mixtures E4M3 |
| PLE n-gram | fp8 fixed-scale, in-container (served `--ngram-placement ram`, same 47.7 GiB pin law) |
| Speculator | the model's own MTP head (card default depth 3; served at 2) |
| Vision | bf16 vision tower, in-container |
**Base model: the OFFICIAL (censored) Qwen3.8-Flash-Next** — the fleet's
only frontier model at base alignment; the abliterated family lives on
frank-dragoon. Card quality: full-vocab teacher-forced KL vs bf16 over
~75K positions — mean 0.0789 all-tokens / 0.0239 assistant turns, top-1
agreement 92.85 %.
## Why this swap
- **Publisher-graded precision**: the w4nl codebook + coordinate-descent
refinement is the engine author's own quality ceiling for this
checkpoint; the card's KL ladder (p99 1.65, p99.9 5.33) is published and
reproducible, where our w4dn8 gate measured against our own reference.
- **Formal parity**: the served file is the one the engine author supports
— decode/prefill behavior on 1.0.13 is their tested path (the card's run
line speaks the 1.0.13-era flags; the `--vram-weights-mib`/`--vram-kv-mib`
pair we ship is accepted by pin 1.0.13 per the 2026-10-06 A/B, so the
shape needed no translation).
- **Cost**: the Swift1.5 finetune's refusal-bending behavior leaves this
endpoint (it returns to the official base alignment); the 4-slot/200k
economics and all shape laws carry over unchanged.
## Shape (unchanged from 2026-10-13)
All of it, verbatim — the payload moved 116,416 → 116,322 MiB, inside the
same budget:
- `--tp 2 --tp-wire wht6`, `--max-num-seqs 3`, `--max-model-len 200000`
(capacity contract 3 × 200,000), `--kv-cache-dtype fp8`
- `--placement expert_tiered --host-pool-mib 44928` + explicit
`--vram-weights-mib 25600 --vram-kv-mib 5120` (30,720 of the engine's
30,969 MiB ceiling; pool = contract + 1.3 %)
- `--num-speculative-tokens 2`, `--max-num-batched-tokens 2048`
(MNBT 2048 law), prefix cache 8 GiB host + 64 GiB disk,
`--ngram-placement ram`, froggeric v22.5 `--override-chat-template`
- TTM preflight floor 150,000 / target 204,800 MiB; pinned ledger
2 × 44 GiB pool + 47.7 GiB PLE + 8 GiB prefix ≈ 147 GiB (see
[[frank-archon-radiance-swift15-w4dn8]] for the ledger arithmetic and
the KV dtype law — both are engine properties, payload-independent)
The publisher's card run line (`--host-pool-mib 12288`,
`--expert-vs-cache-ratio 0.82`, seqs 8, spec 3, ~32 GB host RAM) is their
shape on stock 32 GB cards; the fleet ships its own measured shape against
the same file.
## Cutover record — 2026-10-15
1. `rm` (Frank's one-time authorization): the raw
`ukisai/Swift1.5-Qwen3.8-Flash-Next` BF16 checkpoint (336 GiB, spent
conversion source) and `qwen4exp-fp8-native.rad` (173 GiB, two eras
back) left `/home/dank/models` — 150 GB free → 658 GB free. The fp8-native
payload is **no longer on disk**: rebuild =
`scripts/download-frank-archon.sh` + `scripts/build-frank-archon-rad.sh`.
2. Fetch: `scripts/download-frank-archon-iq4r.sh --fetch` as transient unit
`iq4r-fetch` (detached-job protocol; log
`/home/dank/logs/iq4r-fetch.log`, rc sentinel `…rc`).
3. Deploy: `scripts/deploy.sh dankserver --service frank-archon` — launched
detached (the swap restarts the very endpoint the operator session runs
on; the harness timeout catches the session on return).
4. **Boot ledger (2026-10-15, first boot, pin 1.0.13 — the w4nl codebook
loaded, no pin bump)**: `kv_attn` fp8 page **26.00 KiB** at **607,740
tokens** (151,935 blocks = 3.77 GiB) — byte-identical to the w4dn8 boot,
the sizing law carries; `kv_qsa_bkey` 3.25 KiB at the same token count;
`kv_gdn_state` 54 MiB × 6 slots, `kv_gdn_conv` 1.76 MiB × 3,
`kv_qsa_tail` 19.5 KiB × 6; pool **5.00 GiB**, prefix checkpoints 8 ×
56 MiB; the sizing line reads `kv cache 5.00 GiB … backs 3 sequences all
at 200000 tokens at once, which is the worst case`; samplers print **9
rows** per rank (3 × (1+2), inside the wht6 25-row bound);
`speculation: 'mtp' drafts 2 token(s) a step in 2 serial pass(es)` +
prompt-lookup fallback; slab **17,610 (rank0) / 17,658 (rank1) slots**
(64 shadow) — above the w4dn8 record of 17,548 (ten bf16-protected
experts per layer replace twenty-four, so more of the 4-bit table
resides), pinned pool 7,532/7,484 slots; KV lending **4.22 GiB**
(3,571/3,568 mover slots); arenas 896.02/988.17 MiB charged before the
pair split; prefix cache RECOVERED 900,176 blocks + 126 checkpoint slots
from the 64 GiB disk tier (same arch + tokenizer as w4dn8 — the tier
survived the payload swap); froggeric template 28,234 B; vision encoder
live (≤ 4,194,304 px, 16,384 patches/pass). Health gate passed in ~100 s
(radiance binds the port after the full load).
5. **Vision probe (2026-10-15)**: 300 px solid-red PNG via `image_url` —
reasoning stream: "a solid fill of a deep, saturated red hue". Content
came back EMPTY at `max_tokens 64`: the thinking budget consumed all 64
(`reasoning_tokens: 64`, finish `length`) — fleet law, probe thinking
models with ≥ a few hundred tokens.
6. **e2e gate**: deploy exited DEPLOY INCOMPLETE — 30/33 Playwright green,
3 red (`12-crawl`, `13-sso` search SSO, `16-marreta`) on DANKCLAW
surfaces (vhosts intermittently returned `ERR_CONNECTION_RESET` from the
workstation); frank-archon's own health gate passed. The swap touched no
dankclaw runtime file (only homepage services.json text). Census of the
Caddy/auth stack pending; see `log.md`.
7. **Post-boot measurements (pending)**: decode ladder (99.0 t/s single
@ 43.5k, 93.3 @ 197k, 3-way 113.9/43.3/10.9 = 18.4 t/s aggregate —
these are the **w4dn8** numbers, the record to beat, not this file's),
slab-limited prefill (22208/3,397 → 25600/3,584 t/s A-cold 43.5k), draft
acceptance, and `scripts/probe-frank-archon-kv.py` leg D.
## Rollback
Revert the swap commit (`scripts/frank-archon.env` MODEL_FILE line) +
redeploy: `/home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad`
(116,416 MiB), KLD-gated, same shape — **no longer on disk**. The 2026-10-17
disk purge (Frank's approval, trash-staged then emptied: 763 GiB free after)
removed it; the byte-exact copy lives on the Hub as
`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`
(`qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`)
and re-fetches through `scripts/download-frank-archon-iq4r.sh`'s
hfget/verify pattern retargeted at that repo. The same purge deleted the
`radiance-state` kvcache tiers, `~/.radiance-cache-w4a8-*`,
`~/.cache/frank-primary`, and `/opt/vllm/moremodels` (GLM-4.7 GGUF) — all
derived or retired. The fp8-native era is rebuild-only (see the cutover
record). The w4dn8 runbook keeps every shape law, the KV dtype table, the
slab ladder, and the tuning levers:
[[frank-archon-radiance-swift15-w4dn8]].
## Related
- [[frank-archon-radiance-swift15-w4dn8]] — the replaced payload and the
full shape-law library (slab slope, KV dtype arithmetic, TTM law)
- [[frank-archon-radiance-qwen38-native-fp8]] — the fp8-native era
(rebuild-only now; owns the descriptor-selection law)
- [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here
- [[deployment]] — the service-scoped deploy this cutover rides
@@ -1,6 +1,6 @@
# frank-archon — Radiance (StillDeadcode), Swift1.5 Qwen3.8-Flash-Next W4Dn8, 2× R9700 TP=2
**Status: LIVE — deployed 2026-10-08 (main `3ca64e1`, e2e 33/33).** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 2× Radeon AI PRO R9700 (gfx1201), TP=2
**Status: RETIRED — superseded 2026-10-15 by the publisher iq4r payload.** · Lived 2026-10-08 → 10-15 (main `3ca64e1`, e2e 33/33). · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 2× Radeon AI PRO R9700 (gfx1201), TP=2
The FP8-native era (`qwen4exp-fp8-native.rad`, 173.0 GiB, no-recipe convert)
is replaced by a **GPTQ-calibrated 4-bit container** of
@@ -9,10 +9,10 @@ finetune) converted with the recipe
`scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe` on engine
**1.0.13** (`stilldeadcode/radiance@sha256:4c3f484564…` — same digest as
the FP8-native era; 1.2.0 was used ONLY for the calibration serve).
Same port, alias, container name, systemd unit. Rollback: revert the
`MODEL_FILE` line in `scripts/frank-archon.env` + redeploy —
`/home/dank/models/rad/qwen4exp-fp8-native.rad` stays on disk (trash
policy; 173 GiB, purge when the window closes).
Same port, alias, container name, systemd unit, and every serving argument
carried into the successor: [[frank-archon-radiance-qwen38-iq4r]]. The
`.rad` itself was purged from disk 2026-10-17; the byte-exact payload
lives on the Hub — see Rollback.
## Why this swap
@@ -145,11 +145,18 @@ dtype`) so a future resize cannot silently under-provision the pool.
### Why the weights slab pays — and why that is not pure loss
The two VRAM budgets are one budget: card 31,856 MiB, minus the 1,710 MiB a
boot consumes outside both (driver context, the 990 MiB activation arena,
hardware queues, headroom, step staging), minus 1,024 MiB kept free ⇒
**`weights + kv ≤ 29,120 MiB`**. Paying for a 6,912 MiB pool therefore cuts the
resident slab 26,071 → 22,208 MiB (−14.8 %), and the evicted experts spill
The two VRAM budgets are one budget, and the ceiling is the engine's own — not a
hand-picked reserve. Card 32,624 MiB, and the table that prints at the pair
24,000/5,120 says `3.42 GiB is neither budgeted nor accounted — the driver's
context, the activation arena and --gpu-headroom-mib come out of it; anything
past those is throughput on the floor`, with `card 0 settled at 2.01 GiB free
against 209.19 MiB (the 96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the
static region)`. Claimable reserve above the ENGINE's floor:
2,058 − 209 = 1,849 MiB ⇒ **`weights + kv ≤ 30,969 MiB`**. (The 29,120 bound
written on 2026-10-11 was my own 1 GiB "keep free" reserve, measured off a card
figure of 31,856 MiB; the engine never asked for that reserve and the seam now
derives the number from its table.) Paying for a 6,912 MiB pool therefore cuts
the resident slab 26,071 → 22,208 MiB (−14.8 %), and the evicted experts spill
into the pinned host pool (40,960 → 44,928 MiB per rank; ledger: pinned pool
slots 7,190 → 10,375). The pinned-RAM ledger rises to ≈147 GiB, so the TTM
floor moved 140,000 → 150,000 MiB (the kernel cap is 204,800).
@@ -253,6 +260,240 @@ from the scored configuration — bf16 can only reduce the attention-state
error, and a re-score is a drained-window job if the gate agreement is ever
needed in the record.
## 2026-10-13 — the power failure, the ttm reset, and 4 × 200,000 fp8 KV
### The outage: a refusal, not a crash
Dankserver lost power on 2026-10-12 ~23:17. The box came back; frank-archon
never did. The unit sat in `activating (auto-restart)`, exiting 1 about 50 ms
after starting, with one journal line on repeat:
```text
ERROR: ttm pages cap is 125988 MiB, the pinned ledger needs >= 150000
frank-archon: raise /sys/module/ttm/parameters/pages_limit …
```
**6,919 refusals in ~20 hours.** No container was created (`podman ps -a` showed
the name with a zero start time), so there was no engine, no port, and no engine
log — the signature of a preflight refusal, which every deployer/monitor view
outside the journal renders as an unexplained crash loop.
The cause is a class, not an incident. The pinned-RAM ledger (~147 GiB: 2 × 44 GiB
expert pools + the 47.7 GiB PLE table + the 8 GiB prefix host tier) rides TTM's
shared pages, capped by `/sys/module/ttm/parameters/pages_limit`. **That knob is
runtime state, and its default is half of RAM** — so the manual raise of
2026-10-05 (52,428,800 pages = 200 GiB) died with the power failure and the cap
fell back to 32,253,039 pages = 125,988 MiB, 24 GiB below the floor the serve
script enforces. The fp8-native era had planned the persistent half of this
("add `ttm.pages_limit=52428800` to the kernel cmdline at the next scheduled
reboot", [[frank-archon-radiance-qwen38-native-fp8]]); it was never applied, and
a power failure is not scheduled.
### The repair: the gate raises the knob itself
`scripts/ttm-preflight.sh` (sourced by `serve-frank-archon.sh`) reads the cap as
the MIN over the knobs the running kernel exposes, and when it is below
`FRANK_ARCHON_TTM_MIN_MIB` it writes `FRANK_ARCHON_TTM_TARGET_MIB` (204,800 MiB =
52,428,800 pages) into every writable knob, **re-reads**, and only then refuses.
The unit is root and the knob is a 0644 root-owned module parameter, so the write
is legal; a kernel that refuses the move produces the same named refusal as
before, and a reboot is now a boot rather than a visit to the machine.
| Knob | Power-failure default | After the gate |
|---|---|---|
| `/sys/module/ttm/parameters/pages_limit` | 32,253,039 pages = 125,988 MiB | 52,428,800 pages = 204,800 MiB |
Seam-tested, not inspection-tested: *the ttm gate raises a reset pages cap and
refuses when the write fails* and *the ttm cap counts only the knobs the kernel
exposes* drive the gate against a temp knob directory — a below-floor writable
knob ends at the target and the start passes; a read-only knob leaves the cap
untouched and the start refuses by name; an absent `pages_max` is skipped, never
read as zero (a zero reading would refuse a healthy host).
### The shape: 4 slots × 200,000, fp8 KV (same day, superseded by the re-cuts below)
Answers the open call of 2026-10-11 from the other side: Frank asked for **four**
concurrent 200,000-token conversations with quantized KV.
fp8 is not a preference at four slots, it is the only dtype that affords them.
The pool must back `seqs × max-model-len` (the engine's hard limit), so the
fourth slot is bought in KV, out of the one per-card budget
(`weights + kv ≤ 29,120 MiB`, the bound as it was understood then):
| dtype | pool for 4 × 200,000 | weights slab left |
|---|---|---|
| bf16 | 10,791 + 440 + 448 = 11,679 MiB | 17,441 MiB (−33 % vs the fp8-era 26,071) |
| **fp8 (deployed)** | 5,713 + 440 + 448 = 6,601 MiB | **22,208 MiB — unchanged** |
So `--vram-weights-mib 22208` / `--vram-kv-mib 6912` stay exactly where the
2026-10-11 change put them (pair sum = the measured 29,120 ceiling), the
44,928 MiB/rank pinned host pool is untouched, and the RAM/TTM ledger does not
move. What the fp8 pool buys: `(6912 − 110×4 − 448) × 1024 / 7.3125` =
**843,566 predicted tokens** = four full-height conversations with 5.4 % slack
(the bf16 pool backed 462,752 = two; the 2026-10-08 fp8 pool backed 386,904 =
1.93).
Terms that move with the dtype and the slot count:
- **Per-slot recurrent state** is dtype-independent and lives inside the pool:
+110 MiB per slot, 440 MiB for four (already in the arithmetic above).
- **The fused TP all-reduce** is declared over `seqs × (1 + spec)` rows: 4 × 3 =
**12 rows**, still inside the 25-row bound at which the `wht6` wire is
bit-exact — so four slots cost nothing in wire exactness.
- **Prefix-cache entries shrink** with the attention plane (110.50 KiB at bf16 →
58.50 KiB at fp8), so the 8 GiB host tier parks ~5.8 sessions of 200k where at
bf16 it parked ~2.9 — four live slots plus a parked set fit with no re-size.
- **`models.yml` is untouched**: `contextWindow` quotes the per-request window
(200,000), which did not change; only the concurrency did.
Expectation carried from the 2026-10-11 A/B, whose arms bracket this shape
(same slab 22,208): prefill is set by the slab, not the dtype, so ~3.1k t/s
cold / ~3.3k warm; decode sits in the 94–118 t/s band with draft acceptance
~0.73–0.75. The 4,379 t/s fp8 prefill of the 2026-10-08 control belongs to the
26,071 MiB slab and does not come back by quantizing KV.
### Boot ledger (2026-10-13 19:21, four slots, fp8 — superseded same evening)
The sizing law's prediction, checked against the engine's own arithmetic:
predicted 843,566 tokens, ledger **843,276** (0.03 % block rounding).
| Group | Page / state | Size | Tokens |
|---|---|---|---|
| `kv_attn` | 13 layers, 4-token blocks, **26.00 KiB** (fp8 page) | 5.23 GiB | **843,276** |
| `kv_qsa_bkey` | 13 layers, 4-token blocks, **3.25 KiB** (dtype-fixed) | 669.10 MiB | **843,276** |
| `kv_gdn_state` | 36 layers, 54.00 MiB, **8 slots** (4 × 2 copies) | 432.00 MiB | per-slot |
| `kv_gdn_conv` / `kv_ple_*` | 36 + 1 + 1 layers, 4 slots | 7.91 MiB | per-slot |
`kv cache 6.75 GiB … backs 4 sequences all at 200000 tokens at once, which is
the worst case` — the capacity contract, in the engine's words. The rest of the
boot is the fp8-era picture: `vram_weights` 21.69 GiB filled to 147 KiB (rank 0)
/ 355 KiB (rank 1) of slack, `unclaimed` 2.03 GiB, cards settled at 1.99 /
2.00 GiB free, `mover: 4967 slab slots (64 shadow), 10375 pinned pool slots,
4.24 GiB static vram`, and the half-refund in plain sight: `4967 slots can be
lent out of the KV cache (5.87 GiB)` (the fp8-era 3,543 MiB pool lent 2.69 GiB).
Two ledger lines that confirm the derived consequences: the sampler prints
**12 rows** (4 sequences × (1 + 2 speculative) — inside the 25-row bound where
`wht6` is bit-exact), and the prefix/tier arithmetic runs on the fp8 entry size
(58.50 KiB, half of the bf16 entry).
### The same-day re-cuts: 4 → 3 slots, then the slab takes the card's slack
Frank's call, twice in one evening. Both are one-env-pair changes, and both were
read off the boot ledger rather than assumed.
**4 → 3 slots (22208/6912 → 24000/5120).** Three slots need
3 × 200,000 at fp8 = 4,285 + 330 + 448 = **5,063 MiB** of pool; the 6,912 pool
was backing 858,969 tokens — 43 % more window than the contract asks for.
Cutting it to 5,120 (predicts 608,027; ledger **607,740**, 0.05 %) hands
1,792 MiB to the slab and drops the all-reduce to 9 rows.
**24,000 → 25,600 (the card's settling slack).** The 24,000 boot printed
`card 0 settled at 2.01 GiB free against 209.19 MiB (the 96.00 MiB
--gpu-headroom-mib and 113.19 MiB kept for the static region)` — 1,849 MiB above
the ENGINE's own floor, with the budget table calling the surplus `throughput on
the floor`. Claiming 1,600 MiB of it puts the pair at 30,720 and the ledger at
`unclaimed 477.30 MiB` (cards ~336 MiB free, floor 209.19).
| pair | slab slots (rank 0) | pinned pool slots | A-cold 43.5k | concurrent 197k |
|---|---|---|---|---|
| 22208/6912, seqs 4 | 14,753 | 10,375 | 3,397 t/s | 3,307 t/s (4-way) |
| 24000/5120, seqs 3 | 16,235 | 8,893 | 3,290 t/s | 3,051 t/s (3-way) |
| **25600/5120, seqs 3** | **17,548** | 7,580 | **3,584 t/s** | **3,589 t/s** (3-way) |
| 26071/3543, seqs 3 (2026-10-11 control) | — | 7,190 | 4,379 t/s | — |
The claim shipped with the 25,600 arm — that the slab is a THRESHOLD below
~26 GiB so the extra 1.6 GiB buys ~0 — is **falsified by its own measurement**:
+8.9 % single-stream, +17.6 % concurrent, both outside the ±3 % noise floor. Two
slab points that land inside noise of *each other* (22,208 vs 24,000) are not a
plateau; the pinned host pool is the slow path, so every MiB moved from RAM onto
the card is throughput (the ledger shows the transfer directly: slab
+1,313 slots, pinned pool −1,313).
The cliff is real, it just sits above 25,600: 26,071 measured 4,379 t/s, +22 %
for +471 MiB. At three slots that is out of reach (26,112 + 5,120 = 31,232 > the
30,969 ceiling). At **two** slots the pool falls to 3,072 and the slab reaches
**27,897** — the ceiling exactly, slab above the cliff. That arm is untested and
it is a capacity decision, not a tuning one.
Capacity contract, live-verified on the 3-slot shape: three concurrent prompts
of **199,984 tokens** each (599,952 tokens) were accepted and completed on
`/v1/completions`; a fourth full-height request is refused as "too large to ever
fit" — radiance never truncates.
### The decode ladder (2026-10-13): slots are paid in concurrency, not prefill
Frank's challenge — "why are you not tracking decode?" — was right, and the
answer is a number now. 512 generated tokens (the bench law), greedy, cold
prompts, this shape (3 x 200,000, fp8, slab 25,600):
```
1-stream 43.5k: decode 99.0 t/s (prefill 4,042, accept 0.952)
1-stream 197k: decode 93.3 t/s (prefill 4,826, accept 0.938)
3-stream 43.5k: decode 113.9 / 43.3 / 10.9 t/s wall 83.4s
3-stream 197k: decode 88.9 / 16.3 / 64.6 t/s wall 122.4s
```
Two laws came out of it. **Decode is nearly window-free** — 99.0 at 43.5k,
93.3 at 197k, −6 % for 4.5x the depth — so a long conversation costs
prefill, not tokens-per-second. **Concurrency is where the slot count is
paid**: three simultaneous streams give 18.4 t/s aggregate at 43.5k and 12.5
at 197k against 99 single-stream, and the split is a lottery
(113.9/43.3/10.9), not a fair share — the fused-step schedule hands the step
to whoever it reached first, so one stream finishes at 113 t/s while its
sibling crawls at 10.9. That is what "3 slots" owes per token, and no prefill
number shows it.
Corpus caveat, stated at the claim: this probe's filler is a repetitive
ladder, so MTP acceptance is 0.93–0.97 against the 0.740 of the 2026-10-11
table below; these rows flatter decode relative to that table and are the
baseline for *this* corpus, not a replacement for those rows.
`probe-frank-archon-kv.py` now carries this leg as **D** (leg B generates 24
tokens, which is why a day of shape work priced only prefill).
### Recovery evidence
```text
Oct 13 19:21:08 dankserver ttm-preflight.sh[1623778]: frank-archon: ttm pages cap
is 125988 MiB, below the pinned-ledger floor 150000 — raising the knob to
204800 MiB (52428800 pages)
Oct 13 19:21:08 dankserver ttm-preflight.sh[1623778]: frank-archon: ttm pages cap
204800 MiB clears the pinned-ledger floor 150000
Oct 13 19:21:08 dankserver systemd[1]: Started frank-archon.service
```
`cat /sys/module/ttm/parameters/pages_limit` → 52,428,800 (204,800 MiB) after
the boot, i.e. the root `ExecStartPre` hook did the write the service process
(User=dank) cannot.
### Capacity probe (2026-10-13, live service)
`scripts/probe-frank-archon-kv.py` now reads the slot count, window and KV dtype
from the deployed env on the host, so the probe cannot drift from the service:
it printed `shape: 4 x 200,000 tokens, kv fp8` and fired **four concurrent chats
of 196,960 prompt tokens each** (646,299 chars, 98.5 % of the window), each
carrying a marker whose first word is unique.
| Stream | prompt | wall | prefill | retrieval |
|---|---|---|---|---|
| B1 | 196,961 | 238.3 s | 1,423 t/s | exact, HIT |
| B2 | 196,960 | 98.0 s | 2,043 t/s | exact, HIT |
| B3 | 196,961 | 201.1 s | 993 t/s | exact, HIT |
| B4 | 196,960 | 149.9 s | 1,327 t/s | exact, HIT |
| **total** | **787,842** | **238.4 s** | **3,305 t/s aggregate** | 4/4 |
All four answered their own marker; nothing was preempted, refused, or truncated
(`finish=stop`, `cache_n=0`, so no prefix cache padded the result). Aggregate
prefill equals the single-stream rate measured in the same probe (3,397 t/s at
43.5k), i.e. the four streams share one prefill pipeline at the slab-limited
ceiling and the engine never ran short of KV. Prefix cache intact: the same
43.5k prompt repeated answered in 1.0 s with `cache_n=43008`.
The per-stream `decode` column is NOT a decode rate here — 8 generated tokens
make it first-token dominated (the bench law: generate ≥ 128 tokens to read a
rate). This probe answers one question only: does the endpoint owe four
full-height conversations at once, and answer each one exactly. It does.
## Gates (measured at cutover)
- **KLD gate: PASS (2026-10-08 04:12)** — `kld-gate-swift15.sh`,
@@ -353,15 +594,25 @@ probe; `reasoning_effort` fold; Prometheus target green; `tests/run.sh`
| Layer | Path |
|---|---|
| FP8-native radiance (immediate) | revert `MODEL_FILE` in `scripts/frank-archon.env` + redeploy; `qwen4exp-fp8-native.rad` (173.0 GiB) on disk |
| Publisher iq4r (successor) | [[frank-archon-radiance-qwen38-iq4r]] — the LIVE payload since 2026-10-15 |
| FP8-native radiance | rebuild-only — the `.rad` (173.0 GiB) was `rm`'d 2026-10-15: `scripts/download-frank-archon.sh` + `scripts/build-frank-archon-rad.sh` |
| FreeToken PR #587 | git tag `frank-archon-freetoken-pr587-20261005` |
| tcclaviger vLLM shape G | git tag `frank-archon-vllm-shapeG-20261003` |
Campaign artifacts on disk: BF16 checkpoint (335 GiB), `kld-ref` (24
GiB), grams (`/home/dank/corpora/calib`), corpora JSONL, logs
`/home/dank/logs/swift15_*.log`. The BF16 `.rad` container is deleted
before the final convert (disk gate) — the raw checkpoint is the
re-convert source if the window ever reopens.
**This era's own payload** (`qwen4exp-swift15-w4dn8-i8-hc8m.rad`,
116,416 MiB) was purged in the 2026-10-17 disk cleanup (trash-staged, then
emptied with Frank's approval). The byte-exact container is published as
`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`
(`qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, Hub LFS sha256
`09dd93064365009c508b59012adadab6a36791a50fa319f76b31bf359c01e9ff`) —
re-download restores the exact payload; a full re-convert is no longer
possible without re-fetching a BF16 source (see next paragraph).
Campaign artifacts: the raw BF16 checkpoint was `rm`'d 2026-10-15 (Frank's
one-time authorization), so re-convert needs a fresh BF16 source; `kld-ref`
(24 GiB), grams (`/home/dank/corpora/calib`), corpora JSONL, and the logs
`/home/dank/logs/swift15_*.log` remain. The BF16 `.rad` container was
deleted before the final convert (disk gate).
## Links
@@ -0,0 +1,174 @@
---
type: reference
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork), Bonsai-2-27B PQ2_0, 2 x 200704, q8_0 KV (ACTIVE)"
description: "The ACTIVE frank-zealot runbook (since 2026-10-17) — the PrismML fork build of llama.cpp serving prism-ml/Ternary-Bonsai-2-27B-gguf at PQ2_0 on the RTX 3090: two conversations of 200,704 tokens with q8_0 K + q8_0 V (f16 KV does not fit this card at this window), vision through the publisher's Q8_0 Qwen3-VL projector, MTP impossible (no tensors in the file), and the measured four-arm VRAM census that priced the shape."
tags:
- llm
- frank-zealot
- dankclaw
- llama-cpp
- gguf
- rtx3090
timestamp: 2026-10-17
---
# frank-zealot — llama.cpp (PrismML) Bonsai-2-27B PQ2_0, 2 × 200,704, q8_0 KV
Live since 2026-10-17, replacing the
[[frank-zealot-llama-qwen35-9b-q6k-2x200704]] era (Qwen3.5-9B Q6_K, exact f16
KV, same 2 × 200,704 shape). The trade: the ternary 27B costs ~3.4× the weights
of the 9B and buys a 3× larger, stronger base at the SAME window contract —
paid for by quantizing KV (f16 KV is arithmetically impossible for this
payload on this card, measured below).
## Identity
| | |
|---|---|
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB (24,123 MiB visible to the engine) |
| Endpoint | `http://192.168.0.143:7771` — served model id **`frank-zealot`** (`ALIAS`), `/v1` OpenAI surface |
| Runtime | **PrismML fork build** `/home/dank/GitPackages/llama-prism/llama-server` — vendor release, `--version` = `0.2.0-dev (build 10754, commit 2459f68b5)`, sha256 `12ae5baba033d8cd…`. NOT a preference: see *Why the fork* |
| Unit | root systemd `frank-zealot.service` ([`scripts/frank-zealot.service`](../../scripts/frank-zealot.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
| Serve script | [`scripts/serve-frank-zealot.sh`](../../scripts/serve-frank-zealot.sh), tunables [`scripts/frank-zealot.env`](../../scripts/frank-zealot.env) |
| Weights | `/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-PQ2_0.gguf` — 7,206,168,928 B, sha256 `3907dc1658db1f78a9826bf8d5bcb8dc65db0d466388937af57f2294fae62ec1` |
| Projector | `Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf` — 629,246,976 B (full sha256 pinned in the fetch script), Qwen3-VL tower; **873.10 MiB worst-case VRAM, loaded AFTER the engine's fit check** |
| Fetch | [`scripts/download-frank-zealot-bonsai2.sh`](../../scripts/download-frank-zealot-bonsai2.sh) — `-f`, byte-count + sha256 verified, idempotent, `--verify-only`, no token on the command line |
| Seam | `tests/frank_zealot_unit.bats` (argv assertions against a stub engine) |
## Why the fork — the format is the lock
Read from the file's own metadata, not the model card:
|Key|Value|
|---|---|
|`general.architecture`|`qwen35` (hybrid: Gated DeltaNet + softmax attention)|
|`qwen35.block_count`|64, `full_attention_interval` 4 ⇒ **16 full-attention + 48 DeltaNet layers**|
|Attention|24 heads / **4 KV heads** × key 256 / value 256; `rope.dimension_sections` [11,11,10,0], `freq_base` 1e7, partial rope `dimension_count` 64|
|Widths|embedding 5,120, feed-forward 17,408|
|`qwen35.ssm.*`|conv_kernel 4, state_size 128, group_count 16, **time_step_rank 48**, **inner_size 6,144**|
|Window / vocab|`context_length` 262,144; vocab 248,320 (gpt2 BPE, pre `qwen35`), eos 248046|
|`general.file_type`|**141 = PQ2_0 — 2.13 bpw (group 128)**|
|Tensor histogram|851 tensors: **`pq2_0` ×402** (custom GGML type **142**), f32 ×353, bf16 ×96|
|`prism.hadamard.*`|transform `normalized-sylvester-walsh-hadamard`, block_size 1024, axis `input-last-dimension`, sign_mode explicit, **401 named weights**, sign widths [5120, 6144, 17408], 28,672 sign values, inverse applied on `token_embd.weight`, `gdn_v_grouped` true|
|Sampler metadata|`general.sampling.temp` 1.0 / `top_p` 0.95 / `top_k` 20|
|Chat template|8,952 chars; reads `enable_thinking`, **`reasoning_effort` ∈ {`xhigh` (default), `medium`, `low` — anything else is `raise_exception`**, `preserve_thinking`, `tools`/`tool_call`, image + video content parts|
Stock llama.cpp cannot load this file: type 142 is not in its enum (`gguf-py`'s
stock `GGMLQuantizationType` raises `ValueError: 142 is not a valid…` on the
same read), and even a loader that parsed it would ignore the
`prism.hadamard.*` dequant transform the weights depend on. Same class of lock
as the ROCmFPX custom-type era. The vendor build 10754 already carries the
September upstream fixes (AVX-512 load segfault, and a `GGML_CUDA_FA_ALL_QUANTS`
-capable CUDA backend — proven by q8_0 V serving under flash attention at the
correct 34,816 B/token, below).
## VRAM census (2026-10-17, four arms, production stopped)
The engine's own `common_memory_breakdown_print` ledger + settled `nvidia-smi`,
all arms `--n-gpu-layers 99 --flash-attn on --batch-size 4096 --ubatch-size 2048`,
mmproj attached:
|Shape|KV|projected `model + context + compute`|settled used / free|verdict|
|---|---|---|---|---|
|**2 × 200,704 (pool 401,408)**|**q8_0 / q8_0**|21,439 = 6,539 + 13,627 + 1,272|**22,664 / 1,461**|**DEPLOYED**|
|2 × 200,704|f16 / f16|33,198 = 6,539 + 25,387 + 1,272|—|`cudaMalloc failed: out of memory` on the **25,088.00 MiB** KV request|
|1 × 262,144|q8_0 / q8_0|16,905 = 6,539 + 8,853 + 1,512|18,130 / 5,995|fits, but one slot serializes compaction|
|2 × 262,144|q8_0 / q8_0|≈25,758 (arithmetic, same terms)|—|does not fit (f16 analogue measured 41,118)|
|2 × 200,704|q4_0 / q4_0|15,167 = 6,539 + 7,355 + 1,272|16,392 / 7,733|the roomy fallback rung|
**KV rate law (confirmed to the MiB):** 16 full layers × 4 KV heads × 256 dim =
16,384 elements/token/cache; q8_0 bills 17,408 B per cache ⇒ **34,816 B/token**,
so the 401,408 pool = **13,328.00 MiB** — the ledger printed exactly that
(`CUDA0 KV buffer size = 13328.00 MiB`). f16 doubles it to 25,088 MiB, which is
the wall. The 48 DeltaNet layers bill a fixed recurrent state per **slot**:
context 13,627 − KV 13,328 = **299 MiB for two slots (~150 MiB each)** — pool
arithmetic alone under-prices the shape by that much.
**The projector is outside the fit check.** `[mtmd] estimated worst-case memory
usage of mmproj is 873.10 MiB` is printed before fitting but not billed in it:
settled used (22,664) − projection (21,439) ≈ 1.2 GiB is that projector plus the
CUDA context. This is the difference between "2.6 GiB free on paper" and the
real **1.4 GiB** — the same late-allocation failure class that killed the
DSpark drafter boot in the llama.cpp eras. Any resize must re-read the settled
`nvidia-smi`, never the projection.
## Measured rates (greedy, `Server-Timing`-free `/completion` `timings`)
|Probe|Result|
|---|---|
|Short, warm, single stream (128 tok)|**76.6 t/s** decode (the 9B Q6_K predecessor ran 94.4 — the ternary 27B is ~19 % slower, weight bandwidth + Hadamard dequant)|
|First request after boot|2.6 t/s prefill / 37.7 t/s decode — **warmup artifact; never report the first request** (measured twice on this payload)|
|Prefill at ~35k depth|**1,258.4 t/s** (27.8 s)|
|Prefill at 166,027-token depth|**691.7 t/s** (240.0 s), zero errors, slot intact|
|Two concurrent streams, 256 tok each|**54.7 + 54.7 = 109.4 t/s aggregate** — `--parallel 2` really batches|
|Long-context retrieval (25,960-token prompt, needle at position 180 of 300 passages)|exact: *"Frank keeps the fleet in git, only through it."*|
## Behavior contract (probes, not assumptions)
- **Reasoning** lands in `reasoning_content` (`--reasoning-format deepseek`) ✓.
- **Tool calls**: a `get_weather` request returns `finish_reason: tool_calls`
with `arguments: {"city":"Seattle"}` — valid JSON, no loop, no HTTP 500. The
fork's `KNOWN_ISSUES.md` warning about malformed/looping tool calls did not
reproduce on build 10754 at this shape.
- **Vision**: a 512 px solid-red PNG answers **"Red"** — but ONLY with a
thinking budget ≥ ~512 tokens. At `max_tokens: 64` the request returns
`finish_reason: "length"` with EMPTY `content` and half a thought in
`reasoning_content`: thinking tokens count against the budget, so a
short-budget vision probe reads as "the model is blind". Images do reach the
model (a 512 px image bills ~280 prompt tokens: 322 total vs 42 for the text
alone). Keep the ≥256 px probe rule from the earlier eras.
- **Log noise law**: `/completion` (text context) at `VERBOSITY=4` prints
`failed to parse reasoning content (response_format json_schema is currently
only supported with parse_reasoning=true), content will be stored in
'content' field` (`probe_request.cpp:789`) whenever the model closes a raw
prompt with ``. It is **cosmetic** — the full chain of thought
is in `content`. The OpenAI `/v1/chat/completions` path parses it into
`reasoning_content` correctly, which is the path the harness and every
consumer uses. Do not diagnose this as a fault, and do not drop verbosity
below 4 — the KV/buffer ledger in the journal is what VRAM forensics read.
- **Long raw-prompt completions can come back empty**: at ~26k+ tokens a bare
`/completion` request generated 0–1 tokens with no error (the model emits an
end-of-turn marker immediately, and a text context has no role frame to keep
it talking). Chat traffic is unaffected; nothing in the harness posts raw
`/completion` at depth.
## Ops notes
- **Trial-window law (cost one boot window on 2026-10-17):** a scratch env file
must use the serve script's OWN variable names — `CACHE_TYPE_K`/`CACHE_TYPE_V`,
`BIND_ADDR`, `NGL`. An invented `KV_CACHE_TYPE` is silently swallowed by the
script's `: "${CACHE_TYPE_K:=f16}"` default and the arm boots **f16**, which
for this payload is an OOM that looks like a model problem.
- **Base alignment:** Bonsai-2 is the BASE (censored) Qwen3.8-27B lineage, so a
refusal is possible on a compaction or subagent request. The CRACK re-export
(`dealignai/Bonsai-2-27B-Ternary-CRACK-GGUF`, already on disk at
`/home/dank/models/Bonsai2-CRACK/Bonsai-2-27B-PQ2_0-CRACK.gguf`, sha256
`5b24ea3e…`) is a **one-line `MODEL_FILE` swap**: the header census proves
byte-identical geometry — 851 tensors, the same `{142:402, f32:353, bf16:96}`
histogram, 64 blocks, vocab 248,320, the same 8,952-char template. The VRAM
ledger does not move. If a refusal ever surfaces, that is the fix; there is
also an MTP-bearing CRACK re-export upstream
(`ManniX-ITA/…-CRACK-MTP-GGUF`, 7,657,489,920 B, +451 MB head) if decode ever
needs the drafter — untested here.
- **Compaction hub unchanged:** all three endpoints still route
`compactionModel` here; 200,704 per slot is the fleet's only window that fits
a worst-case compaction input, and `--parallel 2` keeps compaction off the
critical path of a live generation.
- **Harness side** (`~/.omp/agent/models.yml`): `contextWindow: 200704` and
`maxTokens: 16384` unchanged; `extraBody` now quotes the payload's own
1.0 / 0.95 / 20 with `preserve_thinking: true`. The `reasoningEffortMap` fold
became **load-bearing** (the template raises on any level outside
`xhigh|medium|low`) — an unmapped level is a hard 500 on the role's first
request, naming Jinja rather than the model.
- **Rollback:** revert the swap commit. The Qwen3.5-9B Q6_K payload, its mmproj
and the master CUDA build all remain on disk
(`/home/dank/models/Qwen35-9B-HauhauCS-Q6K/`,
`~/GitPackages/llama-master/build-cuda/`), and the retired era keeps its own
page: [[frank-zealot-llama-qwen35-9b-q6k-2x200704]].
## Related
- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — the replaced era (rollback target)
- [[frank-dragoon-flash-server-v2-swift15]] — the abliterated sibling on the iGPU
- [[frank-archon-radiance-qwen38-iq4r]] — the censored-base flagship this payload shares base alignment with
- [[deployment]] — deploy flow and incidents
+76 -31
View File
@@ -2,25 +2,31 @@
**Dankserver runs the radiance engine (StillDeadcode) on frank-archon**
(digest-pinned `stilldeadcode/radiance` image, engine as entrypoint,
driving `podman run` directly, root systemd unit, :7770): the OFFICIAL
`Qwen/Qwen3.8-Flash-Next-FP8` checkpoint at NATIVE block-FP8 precision
(converted no-recipe to `qwen4exp-fp8-native.rad` — see the runbook for
the descriptor-staging law), on BOTH R9700s TP=2 (`--tp-wire wht6`),
experts in a 40 GiB/rank pinned host pool + VRAM cache (ratio 0.82), the
47.7 GiB PLE table pinned to host RAM, fp8 KV, MTP spec decode depth 3,
2 × 200,000 first boot. Rollback tags frank-archon-freetoken-pr587-20261005
and frank-archon-vllm-shapeG-20261003),
driving `podman run` directly, root systemd unit, :7770):
the **publisher's** 4-bit-codebook container of the official
`Qwen/Qwen3.8-Flash-Next` (`qwen3.8-next-flash-fp8-iq4r-moe.rad`, 116,322 MiB,
sha256-verified at fetch — w4nl codebook experts + int8 trunk, PLE n-gram,
MTP head and vision tower in-container), on BOTH R9700s
TP=2 (`--tp-wire wht6`), experts in a 44 GiB/rank pinned host pool + VRAM cache
(explicit `--vram-weights-mib 25600 --vram-kv-mib 5120` = 30,720 of the
30,969 MiB the engine's own budget table makes claimable), the 47.7 GiB PLE
table pinned to host RAM, **fp8 KV at 3 × 200,000**, MTP spec decode depth 2,
froggeric v22.5 template. Rollback: the in-house Swift1.5 w4dn8 `.rad` (on
disk); tags frank-archon-freetoken-pr587-20261005 and
frank-archon-vllm-shapeG-20261003),
and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2
(frank-dragoon, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned
digest serving the Quat3rnion ABLITERATED Swift1.5 finetune of
Qwen3.8-Flash-Next v2, a ~62 GiB v2 .hgn that loads in no other engine,
MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots at 200,192
ctx, vision ON)** and
**llama.cpp CUDA (frank-zealot, port 7771: HauhauCS Gemma4-26B-A4B QAT
Uncensored Q4_K_M GGUF — 26B MoE hybrid (25 sliding-window + 5 full-attention
layers), ONE 200,704-token conversation, MTP disabled,
q8_0 K + q8_0 V (FP8 class), vision via the publisher's own BF16
mmproj, on the RTX 3090)**.
**llama.cpp on the PrismML fork (frank-zealot, port 7771: prism-ml
Ternary-Bonsai-2-27B at **PQ2_0** — 27B qwen35 hybrid (16 full-attention + 48
Gated DeltaNet layers), TWO 200,704-token conversations, MTP disabled,
q8_0 K + q8_0 V (f16 KV does not fit the RTX 3090 at this window), vision via
the publisher's own Q8_0 Qwen3-VL mmproj, on the RTX 3090 — and the custom
tensor type 142 + `prism.hadamard.*` metadata mean no other loader can open the
file)**.
> 2026-08-22: the former `vllmScripts` repository is merged into the
> serverAppConfigs repository. This knowledge base moved from the old repo's
@@ -36,10 +42,10 @@ graph LR
gfx1201, 32 GiB each]
B2[radiance StillDeadcode, digest-pinned image
podman run, frank-archon, port 7770]
C[Qwen3.8-Flash-Next FP8 OFFICIAL, native block-FP8
2x R9700 TP=2 wht6, experts 40 GiB/rank pinned pool + VRAM,
PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 3,
2 x 200000, froggeric v22.5]
C[Publisher qwen3.8-next-flash-fp8-iq4r-moe .rad
2x R9700 TP=2 wht6, experts 44 GiB/rank pinned pool + VRAM,
PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 2,
3 x 200000, froggeric v22.5]
end
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
@@ -102,13 +108,25 @@ frank-zealot, port 7771]
(`slow`/`advisor` → frank-archon) carried forward. Superseded by the
MODEL swap: the abliterated v2 .hgn loads only in flash-server, so the
engine moved with the model
- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — **ACTIVE since 2026-10-07**
(root systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** on the RTX 3090
— TWO 200,704-token conversations (`--ctx-size 401408 --parallel 2`), exact
f16 K + f16 V, vision via the publisher's BF16 mmproj. MTP is OFF because
the file has no MTP tensors at all. Priced from the MiMo Q6_K boots (same
qwen35 geometry, same card, same shape). Supersedes the Gemma4 era (below).
- [[frank-zealot-llama-prism-bonsai2-pq20]] — **ACTIVE since 2026-10-17**
(root systemd unit, port 7771): the **PrismML fork** build of llama.cpp
(vendor 10754, commit `2459f68b5` — the only loader for the payload's custom
tensor type 142 + `prism.hadamard.*` transform) serving
prism-ml/Ternary-Bonsai-2-27B-gguf **PQ2_0** (2.13 bpw) on the RTX 3090 —
TWO 200,704-token conversations (`--ctx-size 401408 --parallel 2`), **q8_0 K
+ q8_0 V** (34 KiB/token over the 16 full layers = 13,328 MiB; the f16 pair
needs 25,088 MiB and dies), settled 22,664 used / 1,461 free, vision via the
publisher's Q8_0 mmproj (+873 MiB, charged AFTER the engine's fit check), MTP
OFF (no MTP tensors). 76.6 t/s single-stream decode, 691.7 t/s prefill at
166k, 109.4 t/s aggregate at two streams. Supersedes the Qwen3.5-9B era below.
- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — **RETIRED 2026-10-07 →
10-17** (payload + master CUDA build on disk as rollback): llama.cpp master
(CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive
**Q6_K** on the RTX 3090 — TWO 200,704-token conversations
(`--ctx-size 401408 --parallel 2`), exact f16 K + f16 V, vision via the
publisher's BF16 mmproj. MTP is OFF because the file has no MTP tensors at
all. Priced from the MiMo Q6_K boots (same qwen35 geometry, same card, same
shape). Supersedes the Gemma4 era (below).
- [[frank-zealot-llama-gemma4-q4km-200704]] — **RETIRED 2026-10-07** (same
day; superseded by qwen35-9b-q6k; payload on disk as rollback): llama.cpp
master (CUDA sm_86) serving HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-
@@ -152,15 +170,42 @@ frank-zealot, port 7771]
target; payload on disk; owns the measured boot-pair table)**: the same
runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower —
104.8 tok/s, 3,571 MiB true free
- [[frank-archon-radiance-swift15-w4dn8]] — **ACTIVE since 2026-10-08**:
- [[frank-archon-radiance-qwen38-iq4r]] — **ACTIVE since 2026-10-15**:
the engine author's published container of the OFFICIAL (base-alignment)
Qwen3.8-Flash-Next — `qwen3.8-next-flash-fp8-iq4r-moe.rad`, 116,322 MiB,
Hub-sha256-verified at fetch: 4-bit codes into a 16-level non-uniform
codebook after a Walsh–Hadamard rotation (GPTQ + 3 coordinate-descent
sweeps, 10M-token calibration), ten bf16-protected experts per layer,
int8 trunk/lm_head, fp8 HC + PLE, MTP head, bf16 vision tower; card KL
0.0789 all-tokens / top-1 92.85 % vs bf16. **MODEL_FILE is the only line
that moved** — the 2026-10-13 shape (25600/5120 pair, 44928 pool, fp8 KV
3 × 200,000, MTP 2, wht6, MNBT 2048, froggeric v22.5) carries verbatim.
Rollback: revert the swap commit (w4dn8 `.rad` stays on disk). The
fp8-native `.rad` and the raw Swift1.5 BF16 source were DELETED from
dankserver 2026-10-15 (Frank's one-time rm, volume for this payload;
rebuild via the fetch + build scripts named in the fp8-native runbook)
- [[frank-archon-radiance-swift15-w4dn8]] — **RETIRED 2026-10-08 → 10-15
(superseded by the publisher iq4r swap; payload on disk as the rollback)**:
Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` on radiance
1.0.13 (recipe + 10M-token calibration + KLD gate vs the BF16 container);
the FP8-native payload remains the rollback. **2026-10-11: the KV cache is
UNQUANTIZED (bf16) at 2 x 200,000** — pool 3543 → 6912 MiB (386,904 →
462,752 tokens), weights slab 26071 → 22208, host pool 44928,
`--max-num-seqs 2`. bf16 costs 1.89x fp8 per token (the QSA block key is
dtype-fixed, so it is not a clean 2x); measured price is prefill (-29%),
NOT decode, and spec-decode acceptance rose 0.661 → 0.740.
the FP8-native payload remains the rollback. **2026-10-13: fp8 KV, and the
pair re-cut twice** — `--kv-cache-dtype fp8` first at
`--max-num-seqs 4`/6912 MiB pool (843,566 tokens, slab 22208), then **3 slots
with the pool cut to the contract (5120 MiB, 608,027 tokens) and the slab
raised 22208 → 24000 → 25600**, the last 1,600 MiB claimed out of the card's
settling slack: the engine's ceiling is **30,969 MiB**, not the 29,120 my
1 GiB reserve invented. fp8 is what funds the slots — the pool must back
`seqs × window`, and at bf16 four slots cost 11,679 MiB (slab 17,441, −33 %).
Measured on the slab: A-cold 43.5k prefill 22208 → 3,397, 24000 → 3,290,
**25600 → 3,584 t/s**; 3-way 197k aggregate 3,051 → **3,589 t/s**. A shipped
prediction that the slab was a threshold below 26 GiB died on its own
measurement — it is a slope, and 26,071 (4,379 t/s) is the cliff. The bf16
arm (2 × 200,000, 462,752 tokens, prefill −29 %, acceptance 0.661 → 0.740)
and the 4-slot fp8 shape are the documented rollbacks.
**2026-10-13 also: the pinned-ledger preflight now raises `ttm.pages_limit`
itself** — the 2026-10-12 power failure
reset the knob to its default (125,988 MiB) and the endpoint refused to boot
6,919 times in ~20 h; `scripts/ttm-preflight.sh` is the repair.
- [[frank-archon-radiance-qwen38-native-fp8]] — **RETIRED 2026-10-08 (the
w4dn8 swap replaced it; its `.rad` stays on disk as the rollback)**
(root systemd unit, port 7770): the StillDeadcode **radiance** engine
+19 -4
View File
@@ -182,12 +182,20 @@ The LLM inference fleet on the two hosts. Full index:
### Deployment and Topology
- [[deployment]] — unified deploy and update process for both servers
- [[frank-archon-radiance-swift15-w4dn8]] — STAGED frank-archon runbook:
- [[frank-archon-radiance-qwen38-iq4r]] — **ACTIVE frank-archon runbook
(2026-10-15)**: the PUBLISHER's container of the official
Qwen3.8-Flash-Next (`qwen3.8-next-flash-fp8-iq4r-moe.rad`, 116,322 MiB,
w4nl 4-bit-codebook experts + int8 trunk, Hub-sha256-verified at fetch)
swapped in as a MODEL_FILE-only change — the 2026-10-13 shape (slab
25600 / fp8-KV 5120, 3 × 200,000, host pool 44,928, MTP depth 2) carries
verbatim; rollback = revert the swap commit (the w4dn8 `.rad` stays on
disk; the fp8-native `.rad` was deleted 2026-10-15, rebuildable)
- [[frank-archon-radiance-swift15-w4dn8]] — RETIRED 2026-10-08 → 10-15 frank-archon runbook:
Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` (radiance
1.0.13), gated on KLD ≤ 0.03 vs the BF16 container and decode ≥ 30.8 t/s;
activates by flipping `MODEL_FILE` in `scripts/frank-archon.env`, rollback
= revert that line (the FP8-native `.rad` stays on disk)
- [[frank-archon-radiance-qwen38-native-fp8]] — the ACTIVE frank-archon
- [[frank-archon-radiance-qwen38-native-fp8]] — the RETIRED 2026-10-05 → 10-08 frank-archon
runbook (2026-10-05): the StillDeadcode radiance engine (digest-pinned
287 MiB image, libr4d gfx1201 kernels) serving the OFFICIAL
`Qwen/Qwen3.8-Flash-Next-FP8` at native block-FP8 (no-recipe convert
@@ -380,8 +388,15 @@ The LLM inference fleet on the two hosts. Full index:
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — the **ACTIVE** frank-zealot
runbook since 2026-10-07 (late): llama.cpp master (CUDA sm_86) serving
- [[frank-zealot-llama-prism-bonsai2-pq20]] — the **ACTIVE** frank-zealot
runbook since 2026-10-17: the PrismML vendor llama.cpp build 10754 (the only
loader for GGML type 142 / `prism.hadamard.*`) serving
prism-ml/Ternary-Bonsai-2-27B-gguf **PQ2_0** on the RTX 3090 — TWO
200,704-token conversations, q8_0 K + q8_0 V (f16 OOMs at 2×200,704: 34 KiB
vs 64 KiB per token over 16 full-attention layers), MTP **off** (no head in
the file), vision through the Q8_0 mmproj; boot 22,664 used / 1,461 free
- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — RETIRED 2026-10-17 (rollback
target, payload on disk): llama.cpp master (CUDA sm_86) serving
HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** on the RTX 3090 —
TWO 200,704-token conversations, exact f16 K + f16 V, MTP **impossible**
(the file carries no `nextn`/`mtp` tensors), vision through the publisher's
+245
View File
@@ -1,5 +1,99 @@
# Bundle Change Log
## 2026-10-13
- **frank-archon outage: a kernel knob, not the model.** The 2026-10-12 23:17
power failure on dankserver reset `/sys/module/ttm/parameters/pages_limit`
from the manual 200 GiB raise back to its default (half of RAM = 125,988 MiB),
below the ~147 GiB pinned-RAM ledger the engine needs; the serve script's
preflight named the refusal and exited 1, and `Restart=always` re-ran it
**6,919 times in ~20 h** — `activating (auto-restart)`, exit 1 in 50 ms, no
container, no engine log. Law adopted: **a sysfs raise is a countdown to the
next reboot**, so the gate raises the knob itself. New
`scripts/ttm-preflight.sh` is sourced by the serve script (verify, raise,
re-verify, then refuse by name) and is also the unit's
`ExecStartPre=+…--raise` root hook — `+` because the service is `User=dank`
and the knob is root-owned 0644, which is what the first attempt (a plain
write inside the serve script) got wrong. `FRANK_ARCHON_TTM_TARGET_MIB=204800`
records the host ceiling.
- **Shape: 4 × 200,000 with fp8 KV (Frank's call).** `--max-num-seqs 4`,
`--kv-cache-dtype fp8`, pool and slab untouched (6,912 / 22,208 MiB = the
measured 29,120/card ceiling): the pool must back `seqs × max-model-len`, and
at bf16 four slots cost 11,679 MiB, which would cut the resident expert slab
to 17,441 (−33 %) — fp8 costs 6,601 and leaves it alone. 4 × (1 + spec 2) = 12
fused-all-reduce rows, inside the 25-row bound where `wht6` is bit-exact.
`models.yml` needed no change: the per-request window is unchanged, only the
concurrency.
- **Deployed at `bae78e7`, gates green.** Boot ledger: `kv cache 6.75 GiB …
backs 4 sequences all at 200000 tokens at once` — 843,276 tokens against the
sizing law's 843,566 (0.03 %), sampler at 12 rows, weights filled to 147 KiB
of slack, cards ~2 GiB free, KV lending 5.87 GiB to the slab. Live probe
(now reading the shape from the deployed env): four concurrent 196,960-token
chats, **4/4 exact marker retrieval**, 787,842 tokens in 238.4 s = 3,305 t/s
aggregate prefill, no preemption; prefix cache verified (repeat answered in
1.0 s, `cache_n=43008`). Playwright e2e 32 passed + 1 flaky (the known Plex
homepage retry); archon bats seam 21/21 including three new ttm-gate tests.
- **Re-cut twice the same evening (Frank's calls), each read off the ledger.**
4 → 3 slots, pool 6,912 → 5,120: three slots need 5,063 MiB and the 6,912 pool
was funding 858,969 tokens — 43 % over the contract. Then slab
24,000 → 25,600, claiming 1,600 of the 1,849 MiB the engine leaves above its
OWN settle floor (`card 0 settled at 2.01 GiB free against 209.19 MiB (the
96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the static region)`): the
real ceiling is 30,969 MiB, not the 29,120 my 1 GiB "keep free" reserve
invented on 2026-10-11. The pair now spends 30,720 and settles
`unclaimed 477.30 MiB` (cards ~336 MiB free).
- **The prediction shipped with that arm was falsified by its own measurement,
which is the whole point of banking the A/B.** Claim: the slab is a THRESHOLD
below ~26 GiB (22,208 → 3,397 and 24,000 → 3,290 t/s are inside noise of each
other), so 1.6 GiB buys ~0. Measured at 25,600: **3,584 t/s** single-stream
cold at 43.5k (+8.9 %) and **3,589 t/s** 3-way aggregate at 197k (wall
193.7 → 164.5 s, +17.6 %) — slab 16,235 → 17,548 slots while the pinned host
pool hands the same experts back (8,893 → 7,580). The slab is a SLOPE; two
points inside noise of each other are not a plateau. The cliff is above
25,600: 26,071 measured 4,379 t/s (+22 % for +471 MiB), unreachable at three
slots (26,112 + 5,120 > the ceiling) and free at two.
- **Capacity verified live at 3 slots:** three concurrent 199,984-token prompts
accepted and completed (599,952 tokens resident), ledger
`backs 3 sequences all at 200000 tokens at once` at 607,740 tokens against the
law's 608,027. A fourth full-height request is refused as too large to ever
fit — radiance never truncates. Probe law: the oversized-prompt counter is
FREE (the refusal fires when PROMPT tokens alone exceed the window —
`prompt: 202,777 prompt tokens does not fit the 200000 token context`) and the
window is ~514,286 characters at the measured 2.5714 chars/token, so a
calibration probe below that pays a real prefill instead of reporting.
- **Open call for Frank:** two slots (pool 3,072 + slab 27,897 = the ceiling
exactly) crosses the cliff for the +22 %, and costs the third concurrent 200k
conversation. Untested arm; capacity decision, not tuning.
- Seam: the pair ceiling is derived from the engine's budget table (30,969) and
owns that arithmetic alone; the KV-pool contract (seqs × window at the dtype's
measured rate + 110 MiB a sequence + 448 MiB of prefix checkpoints) keeps its
own test. 371/371 tests and e2e 33/33 on both deploys.
- **Power-failure casualties, NOT fixed (outside the archon window):**
`mnt-md0.mount` failed at boot, so `/mnt/md0` is an empty mountpoint (the
raid6 array itself is healthy, 7/7; fstab has `nofail` so boot continued) —
media libraries on it read empty. `frank-job-digest.service` (frankbook
job-hunt digest) is in `list-units --state=failed`.
- **The slab is a threshold, not a slope** — measured, same method (probe A
leg: 43,512 tokens, greedy, cold, disk prefix cache clean on every boot):
22,208 → 3,397 t/s, 24,000 → 3,290, **25,600 → 3,290**. The third step
bought nothing; the +22 % belongs to the 2026-10-11 arm at slab 26,071
(4,379 t/s), so the cliff is real and sits at ~26 GiB. The 2-slot arm that
would clear it (pool 3,072 + slab 27,897) remains untested — and the
paragraph above is amended: that arm's payoff is +22 % prefill at 2 slots,
not the +8.9 % the deployed pair ever showed (24,000 → 25,600 = 0 %).
- **Frank's decode challenge, answered with numbers**: 512 tokens, greedy,
cold — **decode 99.0 t/s single-stream at 43.5k and 93.3 at 197k** (decode
is nearly window-free: −6 % for 4.5× depth), while **3 concurrent streams
give 113.9/43.3/10.9 t/s = 18.4 t/s aggregate** at 43.5k and 12.5 at 197k.
The slot count is paid in decode concurrency — a fused-step lottery, not a
fair share — which is exactly the axis the shape hunt was blind to (probe
leg B generates 24 tokens). Corpus caveat at the claim: the ladder's filler
is repetitive, acceptance 0.93–0.97 vs the 0.740 of the 2026-10-11 decode
table, so these rows are the baseline for this corpus, not a replacement.
- **Seam:** `probe-frank-archon-kv.py` gains **leg D** (N-stream decode, 512
tokens, per-stream + aggregate) so no future shape change can be priced on
prefill alone.
## 2026-10-10
- **Fleet service rename: the StarCraft names became the SERVICE names.**
The 2026-10-09 wire alias moved onto every repo artifact: units, serve
@@ -1651,3 +1745,154 @@ every restart since the cutover.
[[frank-archon-radiance-swift15-w4dn8]], and the LLM index entry corrected
(the w4dn8 entry said STAGED, the fp8-native one said ACTIVE).
## 2026-10-15 — frank-archon: publisher iq4r payload swap (MODEL_FILE-only)
- Ask: replace the served model with
`StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe` — the engine author's
published rad-convert of the OFFICIAL Qwen3.8-Flash-Next (w4nl codebook
experts, ten bf16-protected/layer, int8 trunk, PLE/MTP/vision in-container,
card KL 0.0789 / top-1 92.85 %) — **no other argument changes**, mindful
the operator session runs on the endpoint being swapped.
- Disk: Frank authorized one `rm` — the spent Swift1.5 BF16 source (336 GiB)
and the two-era-old `qwen4exp-fp8-native.rad` (173 GiB) left
`/home/dank/models`; 150 GB → 658 GB free. fp8-native is now rebuild-only
(fetch + build scripts); rollback for THIS swap is the w4dn8 `.rad`
(on disk) by reverting the commit.
- Fetch: new `scripts/download-frank-archon-iq4r.sh` (hfget, exact bytes
121,969,901,568 + Hub LFS sha256, `--verify-only` default) as transient
unit `iq4r-fetch`; verified, staged sidecar files kept as provenance.
- Shape carried VERBATIM (25600/5120 pair, 44928 pool, fp8 KV 3 × 200,000,
MTP 2, wht6, MNBT 2048, froggeric v22.5). Tests re-keyed
(`frank_archon_unit.bats` pins the new payload id + sha256, keeps the
w4dn8 rollback provenance), homepage card + trio lore + both indexes
updated; new runbook [[frank-archon-radiance-qwen38-iq4r]]. Commits
`6dcf4a5` (swap) + `f44c398` + `c5ac072` (docs), full bats green.
- Cutover self-severed by design (detached `deploy.sh` survived the session
death when frank-archon restarted); engine healthy in ~100 s. First-boot
ledger: w4nl loads on pin 1.0.13 (no pin bump); kv_attn 26.00 KiB fp8 @
607,740 tokens (identical geometry); slab 17,610/17,658 slots per rank
(> w4dn8's 17,548 — fewer bf16-protected experts pack more 4-bit table);
sampler 9 rows; MTP 2/step; prefix tier RECOVERED 900,176 blocks across
the swap (arch+tokenizer unchanged). Vision probe green (reasoning read
the solid-red PNG; the 64-token budget went all to thinking — fleet law).
- e2e gate went red on three dankclaw specs — NOT the swap: dankclaw rebooted
01:41 and the gluetun boot-race dependency-failed `searxng`, `marreta`,
and `marreta-browser` (jobs consumed while gluetun's start job retried;
`Restart=always` never armed because the services never ran). Cascade
restarted; skill codified (`gluetun-dependent-boot-race-recovery`).
Gate rerunning against the repaired fleet; frank-archon's own health gate
passed every deploy. Open: decode/prefill ladders vs the w4dn8 record,
probe leg D — scheduled probe window.
## 2026-10-15 — authentik: two more human accounts (Dankclaw)
- **Live — mitch (pk 22) and eric (pk 23) provisioned via `ak shell` ORM, no
repo objects.** Each `<user>@franksdank.cloud` in the human-user template
shape (`is_active=True`, display name, password via `set_password` on stdin
only — never argv, so nothing lands in `ps`), groups `Non-Admin Users` +
`jellyfin_users` + `openwebui_users`; no admin group granted. Verified
through the exact PasswordStage code path (`authenticate()` accepted each
password and rejected a wrong one), `app_entitlements(open-webui)` →
`Open WebUI Users` for both (identical to the ridley precedent), and
`is_superuser=False` / `path=users`. Zero `proxysession` rows for either —
groups were set before any login, so no claim purge was due. Downstream:
Jellyfin visibility rides the LDAP outpost cache (expect lag); OpenWebUI
mints their accounts as role `pending` on first login.
- `/serverAppConfigs-docs/Infrastructure/Authentik.md` — human-user template
census extended with pk 22–23.
## 2026-10-17 — frank-zealot: Bonsai-2-27B PQ2_0 trial + cutover (2 × 200,704, q8_0 KV)
- Ask: evaluate `prism-ml/Ternary-Bonsai-2-27B-gguf` (PQ2_0) as frank-zealot's
payload, and specifically measure **2 × 200,704 with q8_0 K + q8_0 V** on the
RTX 3090. Fetch script `scripts/download-frank-zealot-bonsai2.sh` (exact
bytes + sha256, `--verify-only`) landed first; payload verified at
`/home/dank/models/Bonsai2-PQ2_0/` (7,206,168,928 B + 629,246,976 B mmproj).
- Runtime door: the file is `file_type 141` with 402 tensors of custom GGML
type **142** plus `prism.hadamard.*` metadata — stock llama.cpp refuses it,
so the PrismML vendor build 10754 (`2459f68b5`) is forced, not chosen. A
source build with `GGML_CUDA_FA_ALL_QUANTS=ON` was started and then
cancelled as unnecessary: the vendor build already serves q8_0 V under flash
attention at the exact 34,816 B/token (ledger-confirmed).
- Trial-window law cost one boot: the scratch env invented `KV_CACHE_TYPE`
where the serve script reads `CACHE_TYPE_K`/`CACHE_TYPE_V`, so the script's
`:=f16` default silently won and every arm booted f16. That failed window is
now the measured f16 wall: `cudaMalloc` OOM on a 25,088.00 MiB KV request,
engine fit 33,198 vs 23,558 free.
- Four-arm census (production stopped, trap-guarded restore, `bg_12`):
**2 × 200,704 q8_0 = 22,664 used / 1,461 free (DEPLOYED)**; 1 × 262,144
q8_0 = 18,130 / 5,995 (fits, but serializes compaction); 2 × 262,144 q8_0
≈ 25,758 projected (no fit); 2 × 200,704 q4_0 = 16,392 / 7,733 (fallback
rung). Recurrent DeltaNet state bills ~150 MiB per SLOT (context 13,627 −
KV 13,328 for two slots), which pool arithmetic alone misses.
- Behavior probes green: exact-instruction chat, `reasoning_content` present,
`tool_calls` with valid JSON arguments (no fork `KNOWN_ISSUES` loop/500),
512 px solid-red PNG answered **"Red"**, needle retrieval exact through a
25,960-token prompt, 166,027-token prefill clean in 240.0 s. Rates: 76.6 t/s
single-stream (9B ran 94.4), 1,258 t/s at 35k, 691.7 t/s at 166k,
**109.4 t/s aggregate** at two concurrent streams.
- New fleet laws recorded (AGENTS.md + runbook): the mmproj's 873.10 MiB
worst-case is NOT inside the engine's fit projection; the first request after
a boot is a warmup artifact (2.6 / 37.7 t/s measured); a 64-token thinking
budget makes a thinking model look blind; the Bonsai-2 template reads
`reasoning_effort` (xhigh/medium/low, raises otherwise) so the harness fold
became load-bearing on this endpoint, and it reads `preserve_thinking`.
- Cutover: `scripts/frank-zealot.env` (payload + `CACHE_TYPE_*`=q8_0 + ledger
lore), `scripts/deploy-remote.sh` (runtime/model/mmproj gates re-keyed, fork
hint), `tests/frank_zealot_unit.bats` (payload pair pinned; KV assertions now
env-derived instead of an f16 literal), harness `models.yml` (extraBody 1.0
/0.95/20 + `preserve_thinking`, effort-map comment corrected —
`contextWindow` 200704 and `maxTokens` 16384 unchanged), new runbook
[[frank-zealot-llama-prism-bonsai2-pq20]], indexes re-keyed, 9B era retired
in place. Full bats green (371).
- Base alignment: **Frank chose the OFFICIAL censored payload** — publisher
provenance over refusal-freedom. The 9B it replaces was uncensored, so the
homepage card and its warning contract were re-keyed with it: zealot's card
now carries the base-alignment notice, the shared legal warning belongs to
dragoon alone, and the `homepage_config.bats` equality
(`zealot.warn == dragoon.warn`) retired with the payload it described.
The CRACK re-export (byte-identical geometry per header census) stays on
disk as the one-line `MODEL_FILE` fallback if refusals bite.
- **Deployed 19:44** (`--service frank-zealot --service homepage`): production
boot reproduced the census exactly — 22,664 used / 1,461 free, fit projected
21,439 leaving 2,119 ≥ 1,872, KV 13,328.00 MiB at `n_ctx_seq = 200,704`,
`n_slots = 2`, projector loaded, alias `frank-zealot` on `/v1/models`. Live
probes: exact `pong` + `reasoning_content`, `tool_calls` `{"city":"Seattle"}`,
512 px PNG → **Red**, 75.8 t/s decode (1 % off the trial arm). e2e 33/33.
- Two live-path stragglers the cutover also had to catch: `serve-frank-zealot.sh`
still DEFAULTED to the 9B paths and f16 KV (a defaulted run would have served
the retired payload — now re-keyed to the shipped trio), and the OpenWebUI
`model` row's `meta.description` is live DB state naming the 9B (re-keyed,
backup `webui.db.bak-20261017a`; no `open-webui` restart forced — the cached
list picks it up on the next one, and forcing it logs every family session
out for one cosmetic string).
## 2026-10-17 — dankserver disk purge: Swift w4dn8 rollback + orphan caches (docs re-key)
- Ask: "dump the swift rollback and the GLM GGUF — make sure the model
caches and agenttrash are empty as well." Preflight first: `lsof` empty
on both payloads, serving trio names only the iq4r path, and a podman
layer census proved no deleted path backs an image container (the 121 GB
`16b6663d…` layer is the ENGINE image, not a model blob).
- Staged to `/home/dank/.AGENTTRASH/20261017-dump-payloads-caches`, then
purged with Frank's approval: the Swift1.5 w4dn8 `.rad` (113.6 GiB,
rollback target), `GLM-4.7-Flash-UD-Q5_K_XL.gguf` (21 GiB, `/opt/vllm`
dir removed — last non-iq4r model on the box), `radiance-state` kvcache
tiers (69 GiB + 61 MiB, derived), `~/.radiance-cache-w4a8-*` (11 GiB
stale-pin compile cache), `~/.cache/frank-primary` (4.7 GiB stale-name
cache). Also emptied every `.AGENTTRASH` on the host (14 dirs: dated
batches, quadlet/config holds, both checkouts, `/etc/caddy`). 543 → 763
GiB free.
- Recovery: the w4dn8 payload is byte-exact on the Hub as
`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`
(122,071,973,888 B, LFS sha256 `09dd9306…`) — rollback is now a
re-download, not a local file. Corpora (grams 41 G, kld-ref 24 G,
calibration JSONL) deliberately KEPT: re-convert and KLD-gate inputs.
- Post-checks: frank-archon active, `/health` ok; iq4r serving payload
byte-size intact; live kvcache tier (122 MiB) untouched; no process
references a deleted path.
- Docs re-keyed: AGENTS.md fleet row + frank-archon section + both runbook
index rows; iq4r runbook Rollback rewritten (HF re-fetch procedure — the
fetch script's REPO/RAD_NAME/STAGE/DEST/SIZE/SHA256 constants are
hardcoded, so retargeting at the mirror is a script edit); w4dn8 runbook
header corrected (was still "LIVE", never updated at retirement) and its
Rollback table rebuilt.
+2 -2
View File
@@ -90,14 +90,14 @@
<h2>frank-zealot — the fast utility model</h2>
<ul>
<li><strong>Model:</strong> <a href="https://huggingface.co/HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive" target="_blank" rel="noopener">Qwen3.5-9B-Uncensored</a> (Q6_K) on the RTX&nbsp;3090 — roughly 95 tokens per second.</li>
<li><strong>Model:</strong> <a href="https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf" target="_blank" rel="noopener">Ternary-Bonsai-2-27B</a> (PQ2_0 ternary weights) on the RTX&nbsp;3090 — roughly 77 tokens per second.</li>
<li><strong>Equivalent:</strong> last year's frontier level (2025).</li>
<li><strong>Capacity:</strong> 200k-token context, two submissions at a time; sees images.</li>
<li><strong>Inputs:</strong> text, image.</li>
<li><strong>Best for:</strong> code editor autocomplete, agentic search, RAG, research subagents, translation, roleplay, and summarization — any small task that doesn't need much intelligence, and anything with a picture. This is the model Open&nbsp;WebUI offers to family accounts.</li>
<li><strong>Compaction:</strong> if your agent harness supports setting a compaction model, set this one. A compaction job severely slows the bigger models for far longer than it slows this one.</li>
<li><strong>Supervision:</strong> requires tight human supervision.</li>
<li><strong>Warning:</strong> <details class="key-warn"><summary>&#9888; uncensored model</summary><div class="warn-body">Uncensored model: output can include adult content, content that describes harmful, hateful, or illegal acts, content concerning &ldquo;sensitive offensive cybersecurity topics&rdquo;, and other output that may carry severe legal risk. Take this warning seriously &mdash; don&rsquo;t ruin this server for everybody.</div></details></li>
<li><strong>Warning:</strong> <details class="key-warn"><summary>&#9888; censored base model</summary><div class="warn-body">Base-alignment (censored) model: refusals are possible on sensitive topics, and any model can still be steered into risky output. Uncensored work belongs on frank-dragoon. Take this warning seriously &mdash; don&rsquo;t ruin this server for everybody.</div></details></li>
</ul>
<h2>frank-dragoon — deep reasoning</h2>
+5 -5
View File
@@ -295,13 +295,13 @@
"model": {
"unit": "zealot",
"role": "Frank's utility model",
"hf": "https://huggingface.co/HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive",
"hf": "https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf",
"tagline": "HONOR GUIDE ME!",
"sfx": [
"zealot-honor.ogg"
],
"equiv": "Equivalent to last year's frontier level (2025).",
"spec": "Qwen3.5-9B-Uncensored (Q6_K) on an NVIDIA RTX 3090 — ~95 tokens/s; 200k-token context, two submissions at a time; sees images.",
"spec": "Ternary-Bonsai-2-27B (PQ2_0 ternary weights) on an NVIDIA RTX 3090 — ~77 tokens/s; 200k-token context, two submissions at a time; sees images.",
"inputs": [
"text",
"image"
@@ -314,7 +314,7 @@
]
},
"supervision": "Requires tight human supervision.",
"warn": "Uncensored model: output can include adult content, content that describes harmful, hateful, or illegal acts, content concerning “sensitive offensive cybersecurity topics”, and other output that may carry severe legal risk. Take this warning seriously — don’t ruin this server for everybody.",
"warn": "Base-alignment (censored) model: refusals are possible on sensitive topics, and any model can still be steered into risky output. Take this warning seriously — don’t ruin this server for everybody.",
"compaction": "If your agent harness lets you set a compaction model, set this one: a compaction job on the bigger models blocks them for far longer."
},
"url": "",
@@ -366,13 +366,13 @@
"model": {
"unit": "archon",
"role": "Frank's strongest and fastest model runtime",
"hf": "https://huggingface.co/frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700",
"hf": "https://huggingface.co/StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe",
"tagline": "...Power Overwhelming...",
"sfx": [
"archon-power.ogg"
],
"equiv": "Equivalent to Claude Opus 4.8 / OpenAI-Terra (Summer 2026 frontier intelligence).",
"spec": "Our own build: a GPTQ-w4 quant of Swift1.5-Qwen3.8-Flash-Next, quantized and tuned in-house (radiance, ~114 GiB) on twin AMD R9700s with more than 200 GB of system RAM — 200k-token context, three submissions at a time; sees images.",
"spec": "The engine author's published quant: 4-bit codebook experts + int8 trunk of Qwen3.8-Flash-Next (radiance, ~114 GiB) on twin AMD R9700s with more than 200 GB of system RAM — 200k-token context, three submissions at a time; sees images.",
"inputs": [
"text",
"image"
+133 -28
View File
@@ -3,14 +3,17 @@
#
# This file asserts the RADIANCE shape (StillDeadcode engine, digest-pinned
# stilldeadcode/radiance image, BOTH R9700s TP=2 on the wht6 wire, serving
# the Swift1.5 Qwen3.8-Flash-Next GPTQ-w4 .rad built by
# scripts/convert-swift15-w4dn8.sh). It replaces the FreeToken seam of
# the publisher's 4-bit-codebook .rad of the official Qwen3.8-Flash-Next
# (fetched by scripts/download-frank-archon-iq4r.sh; the in-house
# Swift1.5 w4dn8 .rad of scripts/convert-swift15-w4dn8.sh is the on-disk
# rollback). It replaces the FreeToken seam of
# 2026-10-05 (git tag frank-archon-freetoken-pr587-20261005) and pins what
# flags alone do not say: the served window clears the harness worst case,
# the KV pool backs seqs x max-model-len at the declared cache dtype (an
# engine HARD limit, not a ratio), the per-card VRAM pair stays inside the
# measured card ceiling, the pinned-host ledger has a ttm pages cap floor and
# the serve script ENFORCES it, the served window is what models.yml must
# the pinned-host ledger has a ttm pages cap floor, the serve script ENFORCES
# it and RAISES the knob itself when a reboot reset it (the cap's default is
# half of RAM), the served window is what models.yml must
# quote, the host-networking contract (no --publish), the image gate on a
# name@sha256 ref, the .rad payload gate with its size floor, the
# chat-template override mount, the self-healing law (the run returns before
@@ -24,16 +27,28 @@ load helpers
UNIT="$REPO_ROOT/scripts/frank-archon.service"
SERVE="$REPO_ROOT/scripts/serve-frank-archon.sh"
TTM_LIB="$REPO_ROOT/scripts/ttm-preflight.sh"
ENV_FILE="$REPO_ROOT/scripts/frank-archon.env"
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
DOWNLOAD="$REPO_ROOT/scripts/download-frank-archon-swift15.sh"
DOWNLOAD="$REPO_ROOT/scripts/download-frank-archon-iq4r.sh"
BUILD="$REPO_ROOT/scripts/convert-swift15-w4dn8.sh"
HFGET="$REPO_ROOT/scripts/hfget.sh"
TEMPLATE="$REPO_ROOT/scripts/qwen38-froggeric-v22.5.jinja"
# A PATH with `cat` (the gate reads knobs with it) and NO `sudo`, so the
# privileged fallback cannot rescue a write the test means to fail.
nosudo_bin() {
local bin="$BATS_TEST_TMPDIR/nosudo-bin"
mkdir -p "$bin"
ln -sf "$(command -v cat)" "$bin/cat"
ln -sf "$(command -v bash)" "$bin/bash"
echo "$bin"
}
@test "frank-archon unit, serve script, env, fetch and build scripts and template exist" {
[ -f "$UNIT" ]
[ -x "$SERVE" ]
[ -f "$TTM_LIB" ]
[ -f "$ENV_FILE" ]
[ -f "$DOWNLOAD" ]
[ -x "$BUILD" ]
@@ -120,37 +135,40 @@ TEMPLATE="$REPO_ROOT/scripts/qwen38-froggeric-v22.5.jinja"
grep -q '^FRANK_ARCHON_MAX_MODEL_LEN=200000$' "$ENV_FILE"
ml=$(awk -F= '/^FRANK_ARCHON_MAX_MODEL_LEN=/{print $2}' "$ENV_FILE")
[ "$ml" -ge 177356 ]
# Concurrency: 2 slots x the served window is the capacity contract, and
# the fused TP all-reduce is declared over seqs x (1 + spec) = 6 rows,
# Concurrency: 3 slots x the served window is the capacity contract, and
# the fused TP all-reduce is declared over seqs x (1 + spec) = 9 rows,
# under the 25-row bound at which the wht6 wire stays bit-exact. The
# "seqs 2 cannot boot" law belongs to the retired tcclaviger vLLM fork
# (torch.compile query_start_loc constraint, 2026-10-02); radiance booted
# seqs 2 in the fp8-native era and the 2026-10-06 sweep had seqs 2 and 3
# tied inside the +-3% noise floor.
grep -q '^FRANK_ARCHON_MAX_SEQS=2$' "$ENV_FILE"
# tied inside the +-3% noise floor — slots buy concurrency, not speed.
grep -q '^FRANK_ARCHON_MAX_SEQS=3$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_TP=2$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_TP_WIRE=wht6$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_SPEC_TOKENS=2$' "$ENV_FILE"
# expert_tiered placement: vram_weights + the pinned host pool serve the
# per-rank resident set TOGETHER, so cutting the VRAM slab to pay for the
# KV pool moves the pool with it. KV is UNQUANTIZED bf16 (the engine takes
# KV pool moves the pool with it. KV is QUANTIZED fp8 (the engine takes
# only bf16|fp8 — no per-tensor K/V split), the PLE n-gram table is pinned
# in host RAM per Frank's law.
grep -q '^FRANK_ARCHON_PLACEMENT=expert_tiered$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_HOST_POOL_MIB=44928$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_VRAM_WEIGHTS_MIB=[1-9]' "$ENV_FILE"
grep -q '^FRANK_ARCHON_VRAM_KV_MIB=[1-9]' "$ENV_FILE"
grep -q '^FRANK_ARCHON_KV_DTYPE=bf16$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_KV_DTYPE=fp8$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_NGRAM_PLACEMENT=ram$' "$ENV_FILE"
# Prefix cache tiers (publisher pattern; disk tier sized for 2 slots).
grep -q '^FRANK_ARCHON_PREFIX_HOST_MIB=8192$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_PREFIX_DISK_MIB=65536$' "$ENV_FILE"
# The pinned ledger (2 x pool + 47.7 GiB PLE + prefix host) must be
# under the ttm pages cap, and the env states the floor the serve
# script enforces (>= 140,000 MiB — the current cap ~123 GiB fails it,
# which is the point).
# script enforces (>= 140,000 MiB — the kernel's own default cap, half of
# RAM, fails it, which is the point) plus the value it raises the knob to
# when a reboot reset it (the target must clear the floor with slack).
grep -q '^FRANK_ARCHON_TTM_MIN_MIB=' "$ENV_FILE"
ttmm=$(awk -F= '/^FRANK_ARCHON_TTM_MIN_MIB=/{print $2}' "$ENV_FILE")
ttmt=$(awk -F= '/^FRANK_ARCHON_TTM_TARGET_MIB=/{print $2}' "$ENV_FILE")
[ "$ttmt" -gt "$ttmm" ]
pool=$(awk -F= '/^FRANK_ARCHON_HOST_POOL_MIB=/{print $2}' "$ENV_FILE")
ph=$(awk -F= '/^FRANK_ARCHON_PREFIX_HOST_MIB=/{print $2}' "$ENV_FILE")
# 2 pools + PLE 47.7 GiB (48,824 MiB) + prefix host, rounded up.
@@ -188,15 +206,20 @@ TEMPLATE="$REPO_ROOT/scripts/qwen38-froggeric-v22.5.jinja"
}
@test "the per-card VRAM pair stays inside the measured card ceiling" {
# Measured on Dankserver 2026-10-11: card 31,856 MiB, a boot consumes
# 1,710 MiB OUTSIDE both budgets (driver context, the 990 MiB activation
# arena, hardware queues, headroom, step staging) and the fp8 pair settled
# at 1,379 MiB free. Claiming more than 29,120 MiB in total is an
# allocation failure at graph capture, not a warning — so the KV pool and
# the weights slab are one budget, and this seam is the guard.
# The engine's OWN budget table, not a hand-picked reserve. Boot of
# 2026-10-13 at the pair 24000/5120: card 32,624 MiB (31.86 GiB), budgets
# claimed 29,120, `3.42 GiB is neither budgeted nor accounted — the
# driver's context, the activation arena and --gpu-headroom-mib come out of
# it; anything past those is throughput on the floor`, and `card 0 settled
# at 2.01 GiB free against 209.19 MiB (the 96.00 MiB --gpu-headroom-mib and
# 113.19 MiB kept for the static region)`. Claimable reserve above the
# engine's floor: 2,058 - 209 = 1,849 MiB, so the ceiling is 30,969 MiB.
# The 2026-10-11 bound of 29,120 ("keep >= 1 GiB free", card read as 31,856
# MiB) was MY reserve, which the engine never asked for; the shipped pair
# spends the difference and lands ~458 MiB above the floor.
w=$(awk -F= '/^FRANK_ARCHON_VRAM_WEIGHTS_MIB=/{print $2}' "$ENV_FILE")
kv=$(awk -F= '/^FRANK_ARCHON_VRAM_KV_MIB=/{print $2}' "$ENV_FILE")
[ $(( w + kv )) -le 29120 ]
[ $(( w + kv )) -le 30969 ]
}
@test "the serve script wires the engine flags from the env" {
@@ -272,6 +295,83 @@ TEMPLATE="$REPO_ROOT/scripts/qwen38-froggeric-v22.5.jinja"
grep -q -- '--device /dev/dri' "$SERVE"
}
@test "the ttm gate raises a reset pages cap and refuses when the write fails" {
# The cap is a RUNTIME knob: the kernel default is half of RAM, so every
# reboot resets it (measured: the 2026-10-12 23:17 power failure on
# dankserver took it 200 GiB -> 125,988 MiB, and the preflight refusal was
# re-run 6,919 times by Restart=always over ~20 h). Behaviour required of
# the gate, in both directions:
# knob below the floor + writable -> cap ends at the target, start passes
# knob below the floor + unwritable -> cap stays low, start REFUSES by name
local dir="$BATS_TEST_TMPDIR/ttm"
mkdir -p "$dir"
export FRANK_ARCHON_TTM_SYS_DIR="$dir"
export FRANK_ARCHON_TTM_MIN_MIB=150000 FRANK_ARCHON_TTM_TARGET_MIB=204800
echo 32253039 > "$dir/pages_limit" # 125,988 MiB: the kernel default here
run bash -c '. "$1"; ttm_ensure_cap' _ "$TTM_LIB"
[ "$status" -eq 0 ]
[ "$(cat "$dir/pages_limit")" = "52428800" ] # 204,800 MiB in 4 KiB pages
echo 32253039 > "$dir/pages_limit"
chmod 444 "$dir/pages_limit" # the write refused: cap cannot move
# The refusal case must not have `sudo` on PATH: the gate's privileged
# fallback runs `sudo -n`, and on a passwordless host root can write a 444
# file, which would silently test nothing.
run env PATH="$(nosudo_bin)" bash -c '. "$1"; ttm_ensure_cap' _ "$TTM_LIB"
[ "$status" -eq 1 ]
[[ "$output" == *"ERROR: ttm pages cap is 125988 MiB"* ]]
[ "$(cat "$dir/pages_limit")" = "32253039" ]
}
@test "the ttm cap counts only the knobs the kernel exposes" {
# An absent knob is not a zero: a zero-cap reading would refuse a healthy
# host. The effective cap is the MIN over the knobs that exist; pages_max,
# when a kernel exposes one, is the binding one.
local dir="$BATS_TEST_TMPDIR/ttm2"
mkdir -p "$dir"
export FRANK_ARCHON_TTM_SYS_DIR="$dir"
export FRANK_ARCHON_TTM_MIN_MIB=150000 FRANK_ARCHON_TTM_TARGET_MIB=204800
echo 51200000 > "$dir/pages_limit" # 200,000 MiB, no pages_max knob
run bash -c '. "$1"; echo "$(ttm_cap_mib)"' _ "$TTM_LIB"
[ "$status" -eq 0 ]
[ "$output" = "200000" ]
echo 30000000 > "$dir/pages_max" # 117,187 MiB: now the binding knob
run bash -c '. "$1"; echo "$(ttm_cap_mib)"' _ "$TTM_LIB"
[ "$output" = "117187" ]
chmod 444 "$dir/pages_limit" "$dir/pages_max" # both writes refused
run env PATH="$(nosudo_bin)" bash -c '. "$1"; ttm_ensure_cap' _ "$TTM_LIB"
[ "$status" -eq 1 ]
[[ "$output" == *"ERROR: ttm pages cap is 117187 MiB"* ]]
[ "$(cat "$dir/pages_max")" = "30000000" ]
}
@test "the gate runs as a root pre-start hook, where the knob write is legal" {
# The knob is root-owned 0644 and the service runs as User=dank, so the
# gate CANNOT raise it from the service process. systemd's `+` prefix on
# ExecStartPre runs as root even under User= — the canonical privileged
# path — and the gate's own --raise mode is what that line calls.
grep -q '^ExecStartPre=+/home/dank/serverAppConfigs/scripts/ttm-preflight.sh --raise$' "$UNIT"
# Behaviour of that entry point: sourced env values, knob raised, exit 0.
local dir="$BATS_TEST_TMPDIR/ttm3" envf="$BATS_TEST_TMPDIR/ttm3.env"
mkdir -p "$dir"
printf 'FRANK_ARCHON_TTM_MIN_MIB=150000\nFRANK_ARCHON_TTM_TARGET_MIB=204800\n' > "$envf"
echo 32253039 > "$dir/pages_limit"
run env FRANK_ARCHON_ENV_FILE="$envf" FRANK_ARCHON_TTM_SYS_DIR="$dir" \
bash "$TTM_LIB" --raise
[ "$status" -eq 0 ]
[ "$(cat "$dir/pages_limit")" = "52428800" ]
# Usage is a named failure, not a silent no-op.
run bash "$TTM_LIB"
[ "$status" -eq 2 ]
[[ "$output" == *"--raise"* ]]
}
@test "the serve script clears stale containers and self-heals via podman wait" {
# A crashed unit can leave the container name registered forever after;
# the legacy identities hold the same GPUs and die here too. Then block
@@ -283,18 +383,23 @@ TEMPLATE="$REPO_ROOT/scripts/qwen38-froggeric-v22.5.jinja"
grep -q 'exec podman wait "$FRANK_ARCHON_CONTAINER"' "$SERVE"
}
@test "the model id matches the payload's build and fetch scripts" {
# The payload dir/file are the convert script's output layout; the fetch
# script must name the repo the convert stages from. (FP8-native era,
# the rollback payload, keeps its own trio: build-frank-archon-rad.sh +
# download-frank-archon.sh.)
@test "the model id matches the payload's fetch script (and the rollback's build script)" {
# The deployed payload is the PUBLISHER's .rad: the fetch script names
# the repo, the file, and the Hub's LFS sha256; nothing is converted on
# the host for it. The in-house w4dn8 rollback keeps its own trio
# (convert-swift15-w4dn8.sh + download-frank-archon-swift15.sh, and the
# fp8-native era keeps build-frank-archon-rad.sh +
# download-frank-archon.sh).
grep -q '^FRANK_ARCHON_MODEL_DIR=/home/dank/models/rad$' "$ENV_FILE"
grep -q '^FRANK_ARCHON_MODEL_FILE=qwen4exp-swift15-w4dn8-i8-hc8m.rad$' "$ENV_FILE"
grep -q 'qwen4exp-swift15-w4dn8-i8-hc8m.rad' "$BUILD"
grep -q 'ukisai/Swift1.5-Qwen3.8-Flash-Next' "$DOWNLOAD"
grep -q '^FRANK_ARCHON_MODEL_FILE=qwen3.8-next-flash-fp8-iq4r-moe.rad$' "$ENV_FILE"
grep -q 'StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe' "$DOWNLOAD"
grep -q 'qwen3.8-next-flash-fp8-iq4r-moe.rad' "$DOWNLOAD"
grep -q '0af5e96244e80c21ac8edca1719dae49b8a201b94a2f3994c3e24026ceaa4d20' "$DOWNLOAD"
grep -q 'hfget.sh' "$DOWNLOAD"
# The repo is UNGATED — no token may be required on the fetch path.
! grep -q 'HF_TOKEN' "$DOWNLOAD"
# Rollback provenance: the w4dn8 build script names its payload.
grep -q 'qwen4exp-swift15-w4dn8-i8-hc8m.rad' "$BUILD"
}
@test "no bare HIP-reserved env var is defined" {
+34 -24
View File
@@ -1,13 +1,14 @@
# Seam: the frank-zealot llama.cpp root unit on dankclaw.
# frank-zealot is the RTX 3090 endpoint; since 2026-10-07 it serves
# HauhauCS's Qwen3.5-9B-Uncensored-HauhauCS-Aggressive (Q6_K, qwen35 hybrid:
# 32 layers, 8 full-attention + 24 Gated DeltaNet, no MTP tensors in the file)
# via the CUDA (sm_86) build of llama.cpp master: TWO 200,704-token
# conversations (--ctx-size 401408 --parallel 2), exact f16 K + f16 V under
# flash attention, the publisher's own BF16 vision projector, and MTP off —
# the file carries no nextn/mtp tensors, so there is no in-file head to draft
# with. Log verbosity 4 ships so the VRAM-fit forensics (KV layout, buffer
# sizes, graph capture) land in the journal.
# frank-zealot is the RTX 3090 endpoint; since 2026-10-17 it serves
# prism-ml/Ternary-Bonsai-2-27B-gguf at PQ2_0 — the ternary Bonsai-2-27B, a
# qwen35 hybrid with 64 blocks (16 full-attention + 48 Gated DeltaNet, no
# nextn/mtp tensors in the file) — via the PrismML fork build of llama.cpp
# (vendor build 10754), the ONLY loader that registers the payload's custom
# tensor type 142 and honours its prism.hadamard.* metadata: TWO 200,704-token
# conversations (--ctx-size 401408 --parallel 2), q8_0 K + q8_0 V under flash
# attention (f16 KV does not fit this card at this pool — the boot dies on a
# 25,088 MiB cudaMalloc), the publisher's own Q8_0 vision projector, and MTP
# off. Log verbosity 4 ships so the VRAM-fit forensics land in the journal.
# This seam runs the serve script against a stub engine and asserts on the
# argv it actually execs, so what is pinned is the command the operator gets.
@@ -95,10 +96,15 @@ run_stage() {
grep -q 'FRANK_ZEALOT_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-zealot.env' "$UNIT"
}
@test "the shipped env points at the llama.cpp runtime and the Qwen3.5-9B Q6_K GGUF" {
# The payload path IS the endpoint's identity.
grep -qE '^LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server$' "$ENV_FILE"
grep -qE '^MODEL_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3\.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K\.gguf$' "$ENV_FILE"
@test "the shipped env points at the fork runtime and the Bonsai-2 PQ2_0 pair" {
# The payload path IS the endpoint's identity, and the runtime is part of
# it here: the file's custom tensor type 142 + prism.hadamard.* metadata
# make the PrismML build the only loader, so a retarget that moves the
# model but leaves the master build behind ships an endpoint that cannot
# load at all.
grep -qE '^LLAMA_BIN=/home/dank/GitPackages/llama-prism/llama-server$' "$ENV_FILE"
grep -qE '^MODEL_FILE=/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-PQ2_0\.gguf$' "$ENV_FILE"
grep -qE '^MMPROJ_FILE=/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-mmproj-Q8_0\.gguf$' "$ENV_FILE"
}
@test "the deployer gates on exactly the payload paths the env file ships" {
@@ -135,19 +141,23 @@ run_stage() {
[ $((pool / slots)) -eq 200704 ]
}
@test "the KV contract is exact f16 K + f16 V, flash-attn on" {
@test "the KV dtype pair follows the env file and flash-attn stays pinned on" {
# Asserted against the env rather than a literal so a retune moves one
# file, not two. The shipped pair is q8_0/q8_0: this payload bills
# 16 full-attention layers × 4 KV heads × 256 dim = 16,384 elements per
# token per cache, so f16/f16 would need 25,088 MiB of the 24,123 MiB card
# (measured: cudaMalloc OOM, engine fit 33,198 vs 23,558 free) while q8_0
# bills 34,816 B/token = 13,328.00 MiB at the 401,408 pool. The boot
# ledger, not this file, is the referee. flash-attn is not decoration:
# quantized V is only legal with it, and it must stay explicit so a
# future quant swap cannot silently fall back.
local env_k env_v
env_k="$(sed -nE 's/^CACHE_TYPE_K=(.+)$/\1/p' "$ENV_FILE")"
env_v="$(sed -nE 's/^CACHE_TYPE_V=(.+)$/\1/p' "$ENV_FILE")"
serve_argv
[ "$status" -eq 0 ]
# Frank, 2026-10-07: the 9B rung funds unquantized KV (the 26B MoE era
# shipped q8_0 V because the card was full). qwen35 bills the pool on the
# 8 full-attention layers only (8 × 4 KV heads × 256 dim = 8,192
# elements/token ⇒ 32 KiB/token at f16/f16, 12.25 GiB at the 401,408
# pool); the 24 DeltaNet layers cost a fixed recurrent state per slot.
# The boot ledger, not this file, is the referee. flash-attn stays pinned:
# any quantized-V fallback (q8_0 = this build's FP8 class) requires it, and
# the build carries GGML_CUDA_FA_ALL_QUANTS=ON.
[ "$(argv_value --cache-type-k)" = "f16" ]
[ "$(argv_value --cache-type-v)" = "f16" ]
[ "$(argv_value --cache-type-k)" = "$env_k" ]
[ "$(argv_value --cache-type-v)" = "$env_v" ]
[ "$(argv_value --flash-attn)" = "on" ]
}
+4 -2
View File
@@ -105,8 +105,10 @@ for s in fleet:
assert [s["name"] for s in fleet] == ["frank-zealot", "frank-dragoon", "frank-archon"], \
"card render order must be zealot, dragoon, archon: " + str([s["name"] for s in fleet])
w = {s["name"]: s["model"].get("warn", "") for s in fleet}
assert w["frank-zealot"] == w["frank-dragoon"] != "", "the legal warnings must be the common uncensored text"
assert "legal risk" in w["frank-zealot"].lower() and "everybody" in w["frank-zealot"].lower()
assert "legal risk" in w["frank-dragoon"].lower() and "everybody" in w["frank-dragoon"].lower(), \
"the abliterated endpoint must carry the common legal warning"
assert "censored" in w["frank-zealot"].lower() and "uncensored" not in w["frank-zealot"].lower(), \
"zealot ships the censored base since 2026-10-17: its card warning must not claim uncensored output"
archon = next(s for s in fleet if s["name"] == "frank-archon")
assert "permission" in archon["model"], "archon card needs the ask-Frank permission callout"
PY