Compare commits

..
Author SHA1 Message Date
Frank 39b8396794 governance: bench is removed from the model-swap procedure
Model changes verify by driving the live service — boot ledger,
per-request timings, one real request per harness role. No bench window
is ever scheduled for a swap. The stop-and-mask policy stays for
deliberate standalone bench windows (it is host protection, not a swap
step). Codified in AGENTS.md (policy lead + summary row), Model Swap and
Rollback (No bench step law, §8 close-out source, Related rescope).
2026-10-15 03:07:46 -07:00
Frank a9d45e7012 frank-archon: the slab is a threshold, and decode is where slots cost
Measured, 512 tokens, greedy, cold, at the deployed 3 x 200,000 fp8 shape:
prefill 22,208 -> 3,397, 24,000 -> 3,290, 25,600 -> 3,290 t/s at 43.5k —
the third slab step bought nothing, so the slab is a threshold whose cliff is
the 2026-10-11 arm at 26,071 (4,379), not a slope. Decode ladder: 99.0 t/s
single-stream at 43.5k, 93.3 at 197k (nearly window-free), and 3 concurrent
streams 113.9/43.3/10.9 = 18.4 t/s aggregate — the slot count is paid in
decode concurrency, the axis the shape hunt never measured because probe leg
B generates 24 tokens. probe-frank-archon-kv.py gains leg D (N-stream decode,
512 tokens, per-stream + aggregate) so no resize is priced on prefill alone.
2026-10-13 20:51:32 -07:00
Frank 2581b8d4b6 docs(frank-archon): the slab is a slope, and the ceiling was mine not the engine's
Measured A/B on the shipped pair 25600/5120: A-cold 43.5k prefill 3,290 ->
3,584 t/s (+8.9 %) and the 3-way 197k aggregate 3,051 -> 3,589 t/s (+17.6 %,
wall 193.7 -> 164.5 s). The prediction I shipped with the arm -- slab is a
threshold below ~26 GiB, so 1,600 MiB buys ~0 -- is falsified by the measurement
it asked for. Two slab points inside noise of EACH OTHER (22,208 / 24,000) are
not a plateau. The cliff is above 25,600: 26,071 measured 4,379 t/s.

Ceiling re-derived from the engine's own budget table (card 32,624 MiB, pair at
29,120 settling 2,058 MiB free against its 209.19 MiB floor => 30,969 claimable).
The 29,120 bound was my 1 GiB \`keep free\` reserve; the seam now derives the
number, so the next resize starts from the engine's arithmetic.

Ledger for the record: slab 16,235 -> 17,548 slots while the pinned host pool
hands the same experts back (8,893 -> 7,580) -- the transfer is the mechanism,
the pinned pool is the slow path. Capacity contract live-verified at 3 slots:
3 x 199,984-token prompts accepted (599,952 tokens), fourth refused as too
large to ever fit.

Probe law learned: the oversized-prompt refusal names the prompt's token count
BEFORE prefill (free calibration) and fires on prompt tokens ALONE, not on
prompt + max_tokens; the window is ~514,286 chars at 2.5714 chars/token.
2026-10-13 20:33:02 -07:00
Frank 09a0251d2d llm(frank-archon): claim the card's settling slack, slab 24000 -> 25600
The 2 GiB/card that \`rocm-smi\` shows free is not idle: the engine's budget
table calls it driver context + the 896 MiB activation arena + 96 MiB headroom,
and says the rest is `throughput on the floor`. Its own settle line is the
number that matters — `card 0 settled at 2.01 GiB free against 209.19 MiB (the
96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the static region)` — so
1,849 MiB above the ENGINE's floor is claimable and the real ceiling is 30,969
MiB, not the 29,120 my 1 GiB reserve invented on 2026-10-11.

Shipped pair 25600/5120 = 30,720: +1,600 MiB of slab, ~458 MiB above the floor.
The slab is already saturated at 24,000 (mover took all of it, 451 KiB slack),
so this is the only direction left — KV is at its 3-slot floor (5,063 needed of
5,120).

Named prediction, because today's two points say the slab is a THRESHOLD and not
a slope (22,208 -> 3,397 t/s, 24,000 -> 3,290 t/s at 43.5k single-stream): this
arm buys ~1,600 resident expert slots and ~0 prefill. The point that moves
prefill is 26,071 (2026-10-11: 4,379 t/s), fundable only at two slots. Prefill
moving > +-3 % here falsifies the threshold reading.

Seam: the ceiling gate now derives from the engine's budget table (30,969) and
is the single owner of the pair arithmetic; the duplicate I put in the shape
test is removed.
2026-10-13 20:18:41 -07:00
Frank 066813ea95 llm(frank-archon): 3 x 200,000, pool cut to the contract, slab +1,792 MiB
Frank's call: three slots, and spend the KV the third slot doesn't need on
resident experts. `--max-num-seqs 3`, `--vram-kv-mib 6912 -> 5120`,
`--vram-weights-mib 22208 -> 24000` — the pair is still exactly the measured
29,120 MiB/card ceiling. The pool funds 608,027 tokens against the 600,000-token
contract (3 x 200,000, +1.3 %), and the slab gains 8.1 %.

The direction is the 2026-10-11 A/B: prefill is slab-limited, KV beyond the
contract buys nothing. At fp8 the slab curve measured on this host is
22,208 -> 3,397 t/s and 26,071 -> 4,379 t/s at 43.5k, i.e. ~0.25 t/s per MiB, so
24,000 predicts ~3,850 t/s single-stream. Falsifiable; the probe is the check.

The cost is named, not hidden: a FOURTH full-height conversation no longer fits
(4 x 200,000 needs 6,601 MiB of pool) and is refused as too large to ever fit —
radiance never truncates. Reverting to four slots is one env pair.

Seam: two new arithmetic gates, because both failures are boot-then-400 shape
faults that no flag check catches — the pair must respect the per-card ceiling,
and the pool must back seqs x max-model-len at the dtype's measured rate plus
the dtype-independent terms (110 MiB a sequence of recurrent state, 448 MiB of
prefix checkpoints).
2026-10-13 20:00:39 -07:00
Frank 7daa886dd6 docs(frank-archon): work-log entry and the 4-way capacity probe
Probe on the live service: four concurrent 196,960-token chats, 4/4 exact
marker retrieval, 787,842 tokens in 238.4 s (3,305 t/s aggregate prefill = the
single-stream ceiling, so the four streams shared one pipeline and the KV pool
never ran short); prefix-cache repeat answered in 1.0 s with cache_n=43008.
Recorded with the caveat that 8 generated tokens do not make a decode rate.
2026-10-13 19:30:46 -07:00
Frank bae78e7dae docs(frank-archon): the ttm reset outage, and the 4 x 200,000 fp8 shape
Runbook gains the 2026-10-13 section (symptom signature, the runtime-knob law,
the shape arithmetic, the boot ledger that confirms it: 843,276 tokens =
4 x 200,000, 12 all-reduce rows, 5.87 GiB of KV lent to the slab). AGENTS.md
gains the fleet-row update, the KV dtype law at four slots, the sysfs-raise
lesson under Key Lessons, and the pinned-RAM paragraph now points at
scripts/ttm-preflight.sh as the repair. LLM index entry + topology node match.
2026-10-13 19:28:25 -07:00
Frank a6f55f4cc6 fix(frank-archon): raise the ttm knob from a root ExecStartPre — User=dank cannot write it
a9da65e put the raise inside the serve script, which runs as `User=dank`: the
knob is root-owned 0644, so `-w` is false and the write silently failed — the
gate printed its refusal on a cap it had just failed to move. The privileged
path is systemd's `+` prefix: `ExecStartPre=+scripts/ttm-preflight.sh --raise`
runs as root even under `User=`, before ExecStart, so `Restart=always` can
never outrun it. The lib is now also executable (`--raise` sources the env file
and verifies), and keeps a `sudo -n` fallback for a human running the gate from
a shell.

Seam: the refusal cases run with a PATH that has `cat`+`bash` and NO `sudo`
(root writes a 444 file, so a passwordless dev box would have tested nothing),
and both now assert the gate's own exit 1, not merely non-zero. New seam covers
the unit line and the `--raise` entry point end to end (writable knob -> raised,
no args -> usage exit 2).
2026-10-13 19:20:33 -07:00
Frank a9da6589a6 fix(frank-archon): :" guard needs a space — :"${VAR:?}"` runs :204800
The TTM_TARGET_MIB guard I added on 1a6a78e was written `:"...` (no space), so
bash parsed the expansion as one word and executed `cmd :204800` as the
command `:204800`: exit 127 `command not found` on every start, before the
ttm gate ran. Caught on the deploy: the unit crash-looped 34 times in ~4 min
with exactly that journal line.

probe-frank-archon-kv.py now reads the slot count, window and KV dtype from the
deployed env on the host, so probe B exercises the capacity contract that is
actually live (4 x 200,000 today) instead of a hardcoded pair, and each probe's
marker starts with a unique word because show() scores the hit on that word.
2026-10-13 19:16:48 -07:00
Frank 1a6a78ef73 fix(frank-archon): raise the ttm pages cap in the serve path; reshape to 4 x 200,000 fp8 KV
The 2026-10-12 23:17 power failure reset /sys/module/ttm/parameters/pages_limit
from the manual 2026-10-05 raise (200 GiB) back to the kernel default (half of
RAM = 125,988 MiB), below the pinned-RAM ledger floor of 150,000 MiB. The serve
script's preflight named the refusal and exited 1; Restart=always re-ran it
6,919 times over ~20 hours and frank-archon served nobody.

scripts/ttm-preflight.sh (sourced by the serve script) now RAISES the knob
itself before refusing: the unit is root, the knob is a 0644 root module
parameter, and the cap is a runtime knob whose default is half of RAM — so a
manual sysfs write is not a repair procedure, it is a countdown to the next
reboot. FRANK_ARCHON_TTM_TARGET_MIB=204800 records the host ceiling; the gate
re-reads the cap after writing and only then refuses, so a knob the kernel
will not move is still a named failure.

Shape change (Frank's call): FOUR decode slots x 200,000 tokens with fp8
(quantized) KV. The pool must back seqs x max-model-len, so fp8 is what funds
the fourth slot: at bf16 the four slots cost 11,679 MiB of pool and would cut
the resident expert slab to 17,441 MiB (-33 %); at fp8 they cost 6,601 MiB, so
the pool stays 6,912 MiB (predicts 843,566 tokens), the VRAM pair stays at the
measured 29,120 MiB/card ceiling, the weights slab and the 44 GiB host pool are
untouched, and the fused TP all-reduce rides 4 x (1+2) = 12 rows, inside the
25-row bound where the wht6 wire is bit-exact.
2026-10-13 19:09:58 -07:00
Frank c52ba781e3 docs(frank-archon): record the bf16 KV shape, its measured cost, and the probe
Runbook section `2026-10-11 — unquantized KV at 2 x 200,000`: the dtype law
(only bf16|fp8, no K/V split; the QSA block key is dtype-fixed, which is why
bf16 costs 1.89x and not 2x), the pool formula that reproduces both live
boots within 0.1%, the seqs-3-to-2 provenance (the tcclaviger fork's
torch.compile constraint, not radiance's), the boot ledger, and the A/B.

The A/B, fp8 control arm through a transient FRANK_ARCHON_ENV_FILE drop-in:
prefill -29% (4,379 -> 3,126 cold), TTFT at the full window +25% (the 2 x 197k
pair: 110.5 s -> 138.1 s), decode inside the +-3% band on two of three passes,
draft acceptance UP 0.661 -> 0.740. The price is prefill, not decode. Open
call for Frank: keep exact KV, or take the cheap fp8 2 x 200k fit (pool 3,648).

Also corrects two stale index entries (w4dn8 said STAGED, fp8-native said
ACTIVE), adds the worklog entry, and commits the verification tool as
scripts/probe-frank-archon-kv.py — it carries the probe law that cost two
false negatives: radiance answers in reasoning_content, so a chat probe with a
small budget reads as empty content + finish_reason stop on a healthy server.
Tests 368/368.
2026-10-11 01:36:48 -07:00
Frank 0bf26b0741 llm(frank-archon): unquantized bf16 KV at 2 x 200,000
The KV pool at the fp8 shape backed 386,904 tokens — 1.93 sessions of the
served 200,000 window, so a second full-height conversation preempted to the
host/disk prefix tiers. Moving the cache to bf16 (the engine's only
unquantized rung; `--kv-cache-dtype` takes bf16|fp8 and has no per-tensor
K/V split) doubles what a token costs: 7.3125 -> 14.625 KiB per rank
(kv_attn 26.00 -> 52.00 KiB per 4-token page + kv_qsa_bkey 3.25 -> 6.50).

Pool 3543 -> 6912 MiB (437k tokens, >= 2 x 200,000), and the weights slab
pays for all of it: 26071 -> 22208 MiB, the pair pinned to the measured
card ceiling 29,120 MiB (card 31,856 - 1,710 consumed outside both budgets
- 1,024 free). The resident set that leaves VRAM spills into the pinned host
pool, so 40960 -> 44928 MiB, and the prefix host tier doubles with the entry
size (58.50 -> 117 KiB): 4096 -> 8192 MiB. Pinned ledger ~147 GiB, ttm floor
140000 -> 150000 (cap 200 GiB).

--max-num-seqs 3 -> 2: two slots x the window is the capacity contract, and
the "seqs 2 cannot boot" law is the retired tcclaviger fork's torch.compile
query_start_loc constraint, not radiance's (radiance booted seqs 2 in the
fp8-native era; the 2026-10-06 sweep had 2 and 3 tied inside +-3%). Bonus:
the fused all-reduce is now declared over 2 x 3 = 6 rows, under the 25-row
bound at which the wht6 wire is bit-exact.

Cost, paid twice on this engine: idle KV VRAM is also the expert cache's
spillover (lends 2.69 GiB + 740 MiB of arena), so the pool steals weight
residency from the slab AND the lendable cache. Priors: 2.14 GiB of stolen
weights cost 2.5% decode (MAX_MODEL_LEN 262144), the EVC ladder's other
direction cost 8.9%. Prediction -3..-8% vs the fp8 control (136 t/s at 44.5k,
acceptance 0.7246); the deploy boot ledger and live probes are the referee.

Seams: MAX_SEQS/HOST_POOL/KV_DTYPE/PREFIX_HOST literals follow the shape, and
two new invariants guard it — the KV pool must back seqs x max-model-len at
the declared dtype (an engine hard limit: an over-long request is accepted
then refused as too large to ever fit; the formula reproduces the fp8 boot's
386,904 to within 0.1%), and the VRAM pair must stay inside the ceiling.
2026-10-11 01:01:58 -07:00
Frank 394fd8535b docs(log): record the service-rename cutover outcome and the zealot --metrics gap 2026-10-10 17:17:59 -07:00
Frank 55f87fcdce llm: retire the trio service names — fleet services renamed to frank-archon/frank-dragoon/frank-zealot
The 2026-10-09 wire aliases become the service names everywhere:
systemd units, serve scripts, env files and FRANK_* variables,
deploy-plan/deploy-remote service keys, container names, Prometheus
job names + alert rules, Grafana dashboard queries, bats seams,
runbook filenames, homepage/AGENTS lore, and historical prose.
Rollback tags renamed in lockstep. Residue in the tree: zero.
Retired frank-medium/small/npu names are history and stay.

Known casualty: scripts/patches/*.patch context lines were rewritten;
those retired-era vLLM-fork patches no longer apply byte-exact.
2026-10-10 17:03:14 -07:00
201 changed files with 4038 additions and 2821 deletions
@@ -1,4 +1,4 @@
# Tunables for frank-large (kingjones777/Qwen3.8-Flash-Next-ROCmFP4-
# Tunables for frank-dragoon (kingjones777/Qwen3.8-Flash-Next-ROCmFP4-
# STRIX-GGUF on the kingjones30/ROCmFPX llama.cpp runtime, Dankclaw,
# Strix Halo gfx1151, unified memory).
#
@@ -56,7 +56,7 @@ CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/chat-template.jinja
CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}'
# Alias + network.
ALIAS=frank-large
ALIAS=frank-dragoon
BIND_ADDR=0.0.0.0
LISTEN_PORT=7770
@@ -1,5 +1,5 @@
[Unit]
Description=ROCmFPX (kingjones30 fork) llama.cpp Qwen3.8-Flash-Next ROCmFP4 STRIX 177B MoE (dankclaw, frank-large)
Description=ROCmFPX (kingjones30 fork) llama.cpp Qwen3.8-Flash-Next ROCmFP4 STRIX 177B MoE (dankclaw, frank-dragoon)
After=network-online.target
Wants=network-online.target
@@ -7,8 +7,8 @@ Wants=network-online.target
Type=simple
User=dank
Group=dank
Environment="FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env"
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh
Environment="FRANK_DRAGOON_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-dragoon.env"
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-dragoon.sh
Restart=on-failure
RestartSec=10
+1 -1
View File
@@ -6,7 +6,7 @@
# is reachable on this port regardless of the boot tag. The NPU is a
# single-lane device: generations serialize FIFO, and the first request after a
# model swap pays the ~6-11 s load cost. The NPU draws from the unified RAM
# pool shared with frank-large (iGPU).
# pool shared with frank-dragoon (iGPU).
FLM_MODEL=nuextract3:4b
FLM_HOST=0.0.0.0
FLM_PORT=7771
@@ -1,6 +1,6 @@
#!/usr/bin/env bash
# Serve kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF as
# frank-large via the kingjones30/ROCmFPX llama.cpp runtime on
# frank-dragoon via the kingjones30/ROCmFPX llama.cpp runtime on
# Dankclaw (Strix Halo gfx1151, unified memory).
#
# Runtime: kingjones30/ROCmFPX (fork of charlie12345/ROCmFPX with the
@@ -29,7 +29,7 @@
set -euo pipefail
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-large.env}"
ENV_FILE="${FRANK_DRAGOON_ENV_FILE:-$SCRIPT_DIR/frank-dragoon.env}"
if [[ -f "$ENV_FILE" ]]; then
set -a
# shellcheck disable=SC1090
@@ -44,7 +44,7 @@ fi
: "${MMPROJ_PATH:=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf}"
: "${CHAT_TEMPLATE_FILE:=$SCRIPT_DIR/chat-template.jinja}"
: "${CHAT_TEMPLATE_KWARGS:={}}"
: "${ALIAS:=frank-large}"
: "${ALIAS:=frank-dragoon}"
: "${BIND_ADDR:=0.0.0.0}"
: "${LISTEN_PORT:=7770}"
: "${CTX_SIZE:=401408}"
+1 -1
View File
@@ -4,7 +4,7 @@
# Runs `flm serve <tag>` on the XDNA2 NPU (50 TOPS, Dankclaw). The NPU is a
# single-lane device: generations serialize in FIFO order, so this is a
# background/triage endpoint, not an interactive multi-user chat. The NPU
# draws from the unified RAM pool shared with frank-large (the iGPU
# draws from the unified RAM pool shared with frank-dragoon (the iGPU
# llama.cpp service); see serverAppConfigs-docs/LLM/frank-npu.md
# for the memory budget and the model-selection tradeoff.
#
+197 -119
View File
@@ -9,11 +9,11 @@ services run from `/home/dank/serverAppConfigs` on each host (the 2026-08-22
merge absorbed the former `vllmScripts` repo, history preserved).
- **Dankserver** (192.168.0.235) — media/library stack (RomM, Copyparty,
Calibre-Web, Jellyfin, Plex) plus the `frank-primary` LLM service.
Calibre-Web, Jellyfin, Plex) plus the `frank-archon` LLM service.
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (flash-server
container on the gfx1151 iGPU) and `frank-secondary` (llama.cpp CUDA on the
Alloy, Pi-hole) plus TWO LLM services: `frank-dragoon` (flash-server
container on the gfx1151 iGPU) and `frank-zealot` (llama.cpp CUDA on the
RTX 3090). Caddy on
Dankclaw reverse-proxies both hosts.
- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
@@ -25,9 +25,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-primary` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | ukisai/Swift1.5-Qwen3.8-Flash-Next BF16 (335.3 GiB, 397/397 sha256-verified) converted by `scripts/convert-swift15-w4dn8.sh` to `qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB**): **all 49,152 experts GPTQ w4** (10M-token calibration on engine 1.2.0) + **24 whole bf16-protected experts/layer** (energy-share law), int8 trunk/lm_head, fp8 HC, PLE n-gram fp8 fixed-scale, on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, explicit `--vram-weights-mib 26071 --vram-kv-mib 3543`), **47.7 GiB n-gram copy pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV** (inherited; pool 3.46 GiB), **MTP spec decode depth 2**, `--max-num-seqs 3`, `--max-model-len 200000`, MNBT 2048, Froggeric v22.5 template. KLD-gated (mean KL 0.0131, top-1 97.92 % vs BF16); decode ~190 t/s live (gate bar 30.8). Cutover 2026-10-08 from fp8-native (payload on disk = rollback). | 7770 |
| Dankclaw | `frank-large` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 262,144 ctx** (KV pool 524,288 positions ≈ 14.0 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), harness window 217,088 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold)|7770|
| Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** (7,359,259,008 B; dense `qwen35` hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP tensors in the file**) + the publisher's own **BF16 mmproj** (921,704,448 B) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **exact f16 K + f16 V** under flash attention (32 KiB/token over the 8 full layers = 12.25 GiB pool), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (forced: no `nextn` tensors exist). Priced from the MiMo Q6_K boots (same geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048. Cutover 2026-10-07 (late) from Gemma4-26B-A4B Q4_K_M 1×200,704. All three endpoints route harness **compaction** here. | 7771 |
| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | ukisai/Swift1.5-Qwen3.8-Flash-Next BF16 (335.3 GiB, 397/397 sha256-verified) converted by `scripts/convert-swift15-w4dn8.sh` to `qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB**): **all 49,152 experts GPTQ w4** (10M-token calibration on engine 1.2.0) + **24 whole bf16-protected experts/layer** (energy-share law), int8 trunk/lm_head, fp8 HC, PLE n-gram fp8 fixed-scale, on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **44 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, explicit `--vram-weights-mib 25600 --vram-kv-mib 5120` = 30,720 of the **30,969 MiB** the engine's own budget table makes claimable: card 32,624 MiB, settle floor 209 MiB), **47.7 GiB n-gram copy pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV** (pool 5,120 MiB = the 3-slot contract + 1.3 % — the only dtype that funds 3 × 200,000 without gutting the slab, since bf16 bills 13.8 KiB/token), **MTP spec decode depth 2**, `--max-num-seqs 3` (the capacity contract is 3 × 200,000; 3 × (1+2) = 9 all-reduce rows, inside the 25-row wht6 exactness bound), `--max-model-len 200000`, MNBT 2048, Froggeric v22.5 template. KLD-gated (mean KL 0.0131, top-1 97.92 % vs BF16); decode ~190 t/s live (gate bar 30.8); prefill is **slab-limited and pays for it** — 22,208 → 3,397, 24,000 → 3,290, 25,600 → **3,584** t/s at 43.5k single-stream, 3-way 197k aggregate 3,051 → **3,589** t/s, with a cliff at ~26 GiB (26,071 → 4,379). Cutover 2026-10-08 from fp8-native (payload on disk = rollback); 3-slot fp8-KV shape 2026-10-13, re-cut twice in the slab's favour (22208/6912 at 4 slots → 24000/5120 → 25600/5120); the 4-slot shape (22208/6912, `--max-num-seqs 4`) is the documented rollback. | 7770 |
| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 262,144 ctx** (KV pool 524,288 positions ≈ 14.0 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), harness window 217,088 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold)|7770|
| Dankclaw | `frank-zealot` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** (7,359,259,008 B; dense `qwen35` hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP tensors in the file**) + the publisher's own **BF16 mmproj** (921,704,448 B) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **exact f16 K + f16 V** under flash attention (32 KiB/token over the 8 full layers = 12.25 GiB pool), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (forced: no `nextn` tensors exist). Priced from the MiMo Q6_K boots (same geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048. Cutover 2026-10-07 (late) from Gemma4-26B-A4B Q4_K_M 1×200,704. All three endpoints route harness **compaction** here. | 7771 |
Retired endpoints (`frank-npu`, and every prior runtime) are in
the [runbook index](#runbook-index). The full swap history lives in
@@ -66,14 +66,14 @@ All edits flow through git. Never edit files on a remote machine directly.
./scripts/deploy.sh dankclaw
./scripts/deploy.sh dankserver
./scripts/deploy.sh dankclaw --service searxng
./scripts/deploy.sh dankserver --service frank-primary
./scripts/deploy.sh dankserver --service frank-archon
./scripts/deploy.sh dankclaw --dry-run
```
The planner maps changed paths to services; the remote runner restarts only
the selected service and records its revision outside the checkout.
Documentation-only changes are a no-op. LLM services are selected only by
their own files — every endpoint (`frank-large`/`frank-secondary`/`frank-primary`)
their own files — every endpoint (`frank-dragoon`/`frank-zealot`/`frank-archon`)
by its unit/env/serve trio in `scripts/` — so
unrelated
container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
@@ -85,7 +85,7 @@ restarts each service whose marker is behind HEAD *and* whose paths changed
since that marker — so a backlog accumulated by service-scoped deploys fires
all at once, and one failing health gate aborts the run with every later
service left pending. Name the service unless a drain is what you want
(`./scripts/deploy.sh dankserver --service frank-primary`).
(`./scripts/deploy.sh dankserver --service frank-archon`).
### Deployment completeness (MANDATORY)
@@ -226,20 +226,27 @@ archive.
## Benchmarking Policy (MANDATORY)
**Benchmarking is NOT part of the model-swap procedure** (2026-10-15). A
model change is verified by driving the live service — boot ledger,
per-request `timings`/`Server-Timing`, one real request per harness role
(`serverAppConfigs-docs/LLM/Model Swap and Rollback.md` §6/§8). No bench
window is scheduled for a swap, ever. The rest of this policy governs
*deliberate* bench windows (flag A/Bs, shape sweeps), which are separate
operations on a live host:
**STOP the production service BEFORE benchmarking** — these servers serve
real users, and a bench beside a live service starves the GPU and corrupts
results. Use the runtime's own harness: `llama-bench` for the llama.cpp
endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion` `timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-large and frank-secondary were measured. Rolling your own bench
endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion` `timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-dragoon and frank-zealot were measured. Rolling your own bench
requires asking first.
Scratch ports are not used on LLM hosts — a
test process cannot share the single GPU anyway; validate against the service
itself.
(A gufo-era law; frank-large ran ROCmFPX llama.cpp 2026-10-02 → 10-05, the
(A gufo-era law; frank-dragoon ran ROCmFPX llama.cpp 2026-10-02 → 10-05, the
halogen 27B container 2026-10-05 → 10-06, and is flash-server since, but the
rollback tags keep it live.) **`gufo bench` cannot rank speculative-decode
settings, and will invert the ranking.**
Measured 2026-09-28 on frank-large at depth 32768: its corpus
Measured 2026-09-28 on frank-dragoon at depth 32768: its corpus
reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across
draft widths, i.e. the same highly-predictable text every run, and 93 % of
every proposal accepted. It reported 29–37 t/s decode where real traffic
@@ -264,12 +271,12 @@ one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a
merely wedged kernel usually still answers ARP.
1. Stop and mask (self-healing would restart it):
- Dankclaw `frank-large` (root unit, halogen podman on the iGPU):
`sudo systemctl stop frank-large && sudo systemctl mask frank-large`
- Dankclaw `frank-secondary` (root unit, llama.cpp on the 3090):
`sudo systemctl stop frank-secondary && sudo systemctl mask frank-secondary`
- Dankserver `frank-primary` (root unit, radiance on both R9700s, TP=2):
`sudo systemctl stop frank-primary && sudo systemctl mask frank-primary`
- Dankclaw `frank-dragoon` (root unit, halogen podman on the iGPU):
`sudo systemctl stop frank-dragoon && sudo systemctl mask frank-dragoon`
- Dankclaw `frank-zealot` (root unit, llama.cpp on the 3090):
`sudo systemctl stop frank-zealot && sudo systemctl mask frank-zealot`
- Dankserver `frank-archon` (root unit, radiance on both R9700s, TP=2):
`sudo systemctl stop frank-archon && sudo systemctl mask frank-archon`
2. Bench.
NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in
`/etc/systemd/system/` is itself a symlink into the checkout — mask
@@ -287,14 +294,15 @@ The harness reaches the fleet through `.omp/agent/models.yml` in the
operator's home — `/home/blank/.omp/agent/models.yml` on the dev box, which is
where the harness runs; there is no copy on the servers. Each provider is one
HOST:PORT; each model `id` is the served alias. **Official model ids are
the StarCraft unit names (2026-10-08) and since 2026-10-09 they are also the
ids on the wire:** `frank-archon` → `192.168.0.235:7770`
(`FRANK_PRIMARY_SERVED_NAME`), `frank-dragoon` → `192.168.0.143:7770`
(`HALOGEN_MODEL_ID`), `frank-zealot` → `192.168.0.143:7771` (`ALIAS`). The
SERVICES, units, scripts, env files and runbooks keep the old names
(`frank-primary`, `frank-large`, `frank-secondary`); only the served alias,
the `models.yml`/`config.yml` ids, the homepage cards, the OpenWebUI grants
and the Hermes routing were renamed. Verify a rename on `/v1/models`, never
the StarCraft unit names (2026-10-08), on the wire since 2026-10-09, and
since 2026-10-10 they are also the SERVICE names:** `frank-archon` →
`192.168.0.235:7770` (`FRANK_ARCHON_SERVED_NAME`), `frank-dragoon` →
`192.168.0.143:7770` (`HALOGEN_MODEL_ID`), `frank-zealot` →
`192.168.0.143:7771` (`ALIAS`). The 2026-10-10 cutover renamed the units,
scripts, env files, containers, deploy services and runbooks to match —
the three former service names are fully retired and exist nowhere in
the repo; historical text in runbooks uses the current names. Verify a
rename on `/v1/models`, never
on a chat response: radiance echoes the **requested** model name, so a 200
proves nothing about the served id (llama.cpp and halogen ignore it).
@@ -309,16 +317,16 @@ provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
`models.yml`, then fix every `modelRoles` entry that named the old endpoint —
a role pointing at a provider the file does not define fails at call time. As
of 2026-10-06 **all three endpoints accept images** (`input: [text,
image]`) — `frank-large` returned to vision with the flash-server v2 swap
image]`) — `frank-dragoon` returned to vision with the flash-server v2 swap
(the v2 vision tower loads beside the checkpoint; the text-only halogen
27B window closed 2026-10-06) — while
`frank-primary` takes them natively (radiance runs the checkpoint's
`frank-archon` takes them natively (radiance runs the checkpoint's
27-layer Qwen-VL tower — patching, pad-token expansion and 3-part rotary
are the engine core; `--mm-max-patches` available),
and `frank-secondary` through
and `frank-zealot` through
the publisher's own BF16 mmproj passed as `--mmproj` — the model is natively
multimodal but the Q6_K GGUF carries no vision tensors (same arrangement the
Gemma4 era used). `frank-secondary`'s `contextWindow` is
Gemma4 era used). `frank-zealot`'s `contextWindow` is
**200704** — the PER-SLOT window (`--ctx-size 401408 --parallel 2` since the
2026-10-07 Qwen3.5-9B swap), not the pool and not the model's native 262144:
the harness must compact before llama.cpp answers a longer prompt with a 400.
@@ -333,20 +341,20 @@ endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's
audio encoder, and left again with the Gemma4 retirement on 2026-10-01
(neither the MiMo nor the Qwen3.5 tower carries audio).
### frank-large — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, ACTIVE)
### frank-dragoon — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, ACTIVE)
The **flash-server engine** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`,
digest-pinned, published tag 0.16.2, 3.6 GB image) as a podman container
driven by `scripts/serve-frank-large.sh` under a **root systemd unit**
(`scripts/frank-large.service`, `User=dank`, `Restart=always`,
`ExecStop=podman stop -t 60 frank-large-halogen`, the script runs `podman
driven by `scripts/serve-frank-dragoon.sh` under a **root systemd unit**
(`scripts/frank-dragoon.service`, `User=dank`, `Restart=always`,
`ExecStop=podman stop -t 60 frank-dragoon-halogen`, the script runs `podman
run` FOREGROUND — the container's exit code is the service's exit so
`Restart=always` stays real, and the engine ledger lands on disk) on port
7770 from the Strix Halo iGPU. Engine stdout goes to
`/home/dank/logs/frank_large.log` — **the boot ledger (weights resident, KV
`/home/dank/logs/frank_dragoon.log` — **the boot ledger (weights resident, KV
pool, MTP head, prompt cache, any clamp) lives there, not in the journal**.
Cutover 2026-10-06 from the halogen 27B-dense era
([[frank-large-halogen27b-dflash2]], retired the same window).
([[frank-dragoon-halogen27b-dflash2]], retired the same window).
Serving `Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated`:
`qwen38-flash-next-v2-swift15-abliterated.hgn` (66,687,678,464 B ≈ 62.1
@@ -368,10 +376,10 @@ sidecar is deliberately not fetched).
**Product facts vs the 27B era, both regressions FIXED:** the base is
**abliterated** (the 27B was a censored-base finetune; refusals were an
accepted trade there) and **vision is ON** (the 27B engine silently
ignored its encoder — frank-secondary owned images for exactly one day at this
ignored its encoder — frank-zealot owned images for exactly one day at this
endpoint).
**Engine laws that shape the config** (all in `scripts/frank-large.env`
**Engine laws that shape the config** (all in `scripts/frank-dragoon.env`
with evidence): `HALOGEN_SPEC_ADAPT=0` is MANDATORY — the default adaptive
policy disables the MTP head for whole requests after low acceptance and
HALVES decode; the head stays on and temp-1.0 traffic pays some acceptance
@@ -413,7 +421,7 @@ page cache beside the checkpoint — the card's quickstart arithmetic does
not bill it as resident, but READ the boot ledger and re-size if the
engine prints a clamp.
Fetch: `scripts/download-frank-large-flashnext.sh` into
Fetch: `scripts/download-frank-dragoon-flashnext.sh` into
`/home/dank/models/halogen-flashnext-v2/` (`-f` + exact byte counts + LFS
sha256 + `--verify-only`, no token on the command line; resume goes
against the RESOLVED CDN URL because curl drops `Range` across HF's 302).
@@ -422,18 +430,18 @@ n-gram + vision tower, flat tokenizer with all six files as regular
files) — the 2026-10-01 crash-loop lesson.
Runbook:
`serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md`.
`serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-swift15.md`.
Rollback: revert the swap commit and redeploy — the 27B payload
(`/home/dank/models/halogen-qwen3.8-27b/`), its fetch script and the
halogen 0.1.4 image digest all remain on disk; the ROCmFPX door behind
that (`scripts/build-rocmfpx.sh`, pin `36e9acd`) is untouched.
### frank-secondary — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
### frank-zealot — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
**llama.cpp master** (0.5.0-dev, commit `5fc4f3c`; CUDA sm_86 build
`/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`,
`GGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit**
(`scripts/frank-secondary.service`, `relink_unit`ed into
(`scripts/frank-zealot.service`, `relink_unit`ed into
`/etc/systemd/system/`), serving
`HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive` at **Q6_K**
(7,359,259,008 B, sha256 `c0ba7beb…` computed locally — the Hub carries no
@@ -489,7 +497,7 @@ tokens). With `--parallel 2` a compaction request no longer serializes behind
a live generation (the Gemma4 era's `--parallel 1` law).
Runbook:
`serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md`.
`serverAppConfigs-docs/LLM/frank-zealot-llama-qwen35-9b-q6k-2x200704.md`.
Rollback: revert the swap commit — the Gemma4 payload
(`/home/dank/models/Gemma4-26B-A4B/`), the MiMo Q6_K payload
(`/home/dank/models/MiMo-Q6K/`), the Swift payload
@@ -497,7 +505,7 @@ Rollback: revert the swap commit — the Gemma4 payload
(`/home/dank/models/Ornith-Q8/`) all remain on disk.
### frank-primary — Radiance (StillDeadcode), Swift1.5-Qwen3.8-Flash-Next GPTQ w4, TP=2 (Dankserver, ACTIVE)
### frank-archon — Radiance (StillDeadcode), Swift1.5-Qwen3.8-Flash-Next GPTQ w4, TP=2 (Dankserver, ACTIVE)
The **radiance engine** (StillDeadcode, Apache-2.0; image
`stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned, **pin 1.0.13**
@@ -526,39 +534,98 @@ Rollback: revert the swap commit — the fp8-native `.rad`
**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 3`,
`--max-model-len 200000` (the harness contextWindow must quote this),
`--placement expert_tiered --host-pool-mib 40960` + **explicit VRAM
partition `--vram-weights-mib 26071 --vram-kv-mib 3543`** (the vram-pair
replaced `--expert-vs-cache-ratio`; the pair sizes the fp8-KV shape at the
deployed boot), `--kv-cache-dtype fp8` (inherited from the fp8-native
env), `--num-speculative-tokens 2`,
`--max-num-batched-tokens 2048`, prefix cache 4 GiB pinned host + 64 GiB
disk at `/home/dank/radiance-state`, **`--ngram-placement ram`**, Froggeric
v22.5 chat template vendored (`scripts/qwen38-froggeric-v22.5.jinja`,
`--override-chat-template`). Boot ledger: KV pool 3.46 GiB (~480k fp8
tokens), two expert pools 57.84 GiB, cards settled 1.41 GiB free.
`--placement expert_tiered --host-pool-mib 44928` + **explicit VRAM
partition `--vram-weights-mib 25600 --vram-kv-mib 5120`** (the vram-pair
replaced `--expert-vs-cache-ratio`; the two are ONE budget —
`weights + kv ≤ 30,969 MiB`, derived from the engine's own budget table: card
32,624 MiB, the pair at 29,120 settling 2,058 MiB free against the engine's
209.19 MiB floor (`--gpu-headroom-mib 96` + 113.19 static), so 1,849 MiB is
claimable and the old 29,120 bound was my hand-picked 1 GiB reserve, not a
hardware wall), `--kv-cache-dtype` **`fp8` — quantized KV, restored 2026-10-13
because it is the only dtype that funds the slot count at this window**,
`--num-speculative-tokens 2`, `--max-num-batched-tokens 2048`, prefix cache
8 GiB pinned host + 64 GiB disk at `/home/dank/radiance-state`,
**`--ngram-placement ram`**, Froggeric v22.5 chat template vendored
(`scripts/qwen38-froggeric-v22.5.jinja`, `--override-chat-template`). Boot
ledger (2026-10-13, pair 25600/5120): `kv cache 5.00 GiB … backs 3 sequences
all at 200000 tokens at once, which is the worst case` — `kv_attn` page
26.00 KiB (fp8) at 607,740 tokens (the sizing law predicted 608,027),
`kv_qsa_bkey` 3.25 KiB at the same token count, `kv_gdn_state` 54 MiB × 6 slots
(3 sequences × 2 copies) = 324 MiB; the sampler prints **9 rows**
(3 × (1+2), inside the 25-row `wht6` exactness bound); the slab filled to
563 KiB of slack with **17,548 slots** (16,235 at 24,000, 14,753 at 22,208 —
every 1,792 MiB buys ~1,300 resident experts, and the pinned host pool hands
the same experts back: 8,893 → 7,580 slots); `unclaimed 477.30 MiB`, cards
settled ~336 MiB free against the engine's 209.19 MiB floor, KV lending
4.22 GiB to the expert cache. Prefill tracks this slab only up to a
**threshold**: 22,208 → 3,397 and 24,000 → 3,290 t/s at 43.5k are one band
(±3 %), and the 25,600 arm measured **3,290** — the third step bought nothing
(the 3,584 arm did not reproduce at 3 slots); the cliff is at ~26 GiB
(26,071 → 4,379 on 2026-10-11's arm), so below that the slab is a threshold,
not a slope.
**Decode ladder (2026-10-13, 512 tokens, greedy, this shape)**: 99.0 t/s
single-stream at 43.5k, 93.3 at 197k — decode is nearly window-free (−6 % for
4.5× depth); **3 concurrent streams give 113.9/43.3/10.9 t/s = 18.4 t/s
aggregate**, so the slot count is paid in decode concurrency (a fused-step
lottery), not in prefill; leg D of `probe-frank-archon-kv.py` measures it.
Shape notes carried from the fp8-native era and unchanged: wht6 gains,
MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk
prefix cache, and the rule that a prompt over `--max-model-len` is
refused, never truncated. What changed: the fp8-native era's NVMe expert
streaming (the 3.5 GB/s wall) is gone — the w4 resident set fits the
40 GiB pool, so decode ran 122–173 t/s in the campaign arms and ~190 t/s
on the deployed shape (512-token streamed probe). The campaign also
host pool, so decode ran 122–173 t/s in the campaign arms and ~190 t/s
on the fp8 shape (512-token streamed probe). The campaign also
measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth,
accept 0.7246); production shipped the fp8-KV + spec-2 pair at cutover —
the KLD gate scored at the fp8-KV shape, so the gate and the boot agree.
Disk-tier sizing law: the engine forces ~9.6 GiB of disk-tier weight
buffers + arena onto the weights side, so `weights + kv` is capped near
29.6 GiB/card; boot with the pair, never `--kv-cache-size-mib`.
accept 0.7246).
**The pinned-RAM ledger is why this is a root unit**: 2 × 40 GiB expert
**The KV dtype law (measured 2026-10-11, applied at 4 slots 2026-10-13)**:
`--kv-cache-dtype` takes ONLY `bf16|fp8` — one dtype for the whole cache, so
llama.cpp's exact-K/quantized-V pair (what frank-zealot runs) has NO radiance
equivalent; the rung under bf16 is fp8 on both halves. Measured per rank from
the two boot ledgers: `kv_attn` (13 layers, 4-token blocks) pages 26.00 → 52.00
KiB, while `kv_qsa_bkey` (13 layers) stays 3.25 KiB at BOTH dtypes (the QSA
block key is an index, not attention state) — so a token costs 7.3125 KiB at fp8
and 13.8125 KiB at bf16: **1.89×, not 2×**. Pool sizing must also pay the
dtype-independent terms that live in the same pool (~110 MiB a sequence of
recurrent state + 448 MiB of prefix checkpoints):
`(vram_kv − 110·seqs − 448) MiB / rate` — bf16 at 6,912 gives 462,946 where the
engine reported 462,752 (0.04 % page rounding); fp8 at 6,912 with four slots
gives 843,566, i.e. the deployed capacity contract 4 × 200,000. **That is why
the dtype is fp8: the pool must back `seqs × max-model-len` as a hard limit, and
bf16 at four slots costs 11,679 MiB — the slab would fall to 17,441 MiB (−33 %),
where fp8 leaves it at 22,208.** The cost is paid twice on this engine, and
half-refunded: idle KV VRAM is also the expert cache's spillover, and the bigger
pool *lends* 6.08 GiB to the slab where the 3,543 MiB fp8 pool lent 2.69 GiB.
`--max-num-seqs` rides 4 × (1 + spec) = 12 fused-all-reduce rows, under the
25-row bound where wht6 is bit-exact (the "seqs 2 cannot boot" law belongs to
the retired tcclaviger fork's torch.compile, not radiance). Disk-tier sizing
law: the engine forces ~9.6 GiB of disk-tier weight buffers + arena onto the
weights side, so `weights + kv` is capped near 29.6 GiB/card; boot with the
pair, never `--kv-cache-size-mib`.
Shape notes carried from the fp8-native era and unchanged: wht6 gains,
MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk
prefix cache, and the rule that a prompt over `--max-model-len` is
refused, never truncated. What changed: the fp8-native era's NVMe expert
streaming (the 3.5 GB/s wall) is gone — the w4 resident set fits the
host pool, so decode ran 122–173 t/s in the campaign arms and ~190 t/s
on the fp8 shape (512-token streamed probe). The campaign also
measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth,
accept 0.7246).
**The pinned-RAM ledger is why this is a root unit**: 2 × 44 GiB expert
pools + the 47.7 GiB n-gram copy (`--ngram-placement ram` is pinned RAM
the card reads) + 4 GiB prefix host ≈ **132 GiB** charged against
the card reads) + 8 GiB prefix host ≈ **147 GiB** charged against
`RLIMIT_MEMLOCK` — the TTM floor is set just above it.
`ttm.pages_limit`/`pages_max` cap GPU-mappable pinned pages; the serve
script's TTM preflight (`FRANK_PRIMARY_TTM_MIN_MIB=140000`) refuses to
start until both knobs are raised in the stop window (sysfs write first,
kernel cmdline + reboot only if read-only).
`ttm.pages_limit`/`pages_max` cap GPU-mappable pinned pages, and **the knob is
runtime state whose default is half of RAM**: `scripts/ttm-preflight.sh` (sourced
by the serve script) therefore RAISES it to `FRANK_ARCHON_TTM_TARGET_MIB=204800`
before refusing, because the 2026-10-12 power failure reset it to 125,988 MiB
and the preflight re-refused 6,919 times. A manual sysfs write in the stop
window is a convenience, not the repair; kernel cmdline persistence
(`ttm.pages_limit=52428800`) is the belt to it.
**Contract facts** (source-verified): the HTTP listener binds AFTER the
full load, so a bare `curl /health` gate is honest (`/health` == `/ping`,
@@ -570,11 +637,11 @@ chat format — no parser flags; vision native (the 27-layer Qwen-VL tower,
`--mm-max-patches`); `/metrics` carries vLLM-compatible names +
`radiance:*` families — the Dankclaw Prometheus scrape is re-added.
Runbook: `serverAppConfigs-docs/LLM/frank-primary-radiance-swift15-w4dn8.md`.
Retired fp8-native era: `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md`.
Retired FreeToken era: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`.
Runbook: `serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md`.
Retired fp8-native era: `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-native-fp8.md`.
Retired FreeToken era: `serverAppConfigs-docs/LLM/frank-archon-freetoken-pr587.md`.
Retired vLLM era: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`.
Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-archon-glm53-flash-w4a16.md`.
## Key Lessons Learned
Durable operational lessons from the fleet's history. Full detail in the
@@ -736,11 +803,22 @@ linked runbooks.
- Swap resize ordering: create and `swapon` the replacement BEFORE `swapoff`
of the old one, or live pages must land in RAM that `MemAvailable`
overstates. Dankclaw swap (64 GiB at `/swap.img`, fstab-driven) protects
the co-tenants, not frank-large — pinned weights don't swap, and paging
the co-tenants, not frank-dragoon — pinned weights don't swap, and paging
the KV pool is worse than the OOM it prevents.
- Host swap state is invisible to `deploy.sh` (no repo file) — check
`swapon --show` before reasoning about headroom.
- **Silent RAM corruption outranks every software theory.** frank-primary's
- **A sysfs raise is a countdown to the next reboot.** `ttm.pages_limit` defaults
to half of RAM, so the manual raise to 200 GiB (2026-10-05) vanished with the
2026-10-12 23:17 POWER FAILURE on dankserver: the cap fell to 125,988 MiB,
frank-archon's pinned-ledger preflight (floor 150,000 MiB) refused the start,
and `Restart=always` re-ran that refusal **6,919 times in ~20 h** — a unit in
`activating (auto-restart)` that exits 1 in 50 ms with no container and no
engine log is a PREFLIGHT refusal, not an engine fault. Repo law now:
`scripts/ttm-preflight.sh` (sourced by `serve-frank-archon.sh`) raises the knob
to `FRANK_ARCHON_TTM_TARGET_MIB` itself, re-reads, and refuses by name only
after a write the kernel rejected. Never prescribe a manual sysfs write as the
recovery procedure for a service that can boot before a human reaches it.
- **Silent RAM corruption outranks every software theory.** frank-archon's
~40k-token output corruption (2026-10-04/05, repetition/garbage in long
agent conversations) was marginal DDR5 at 4800 MT/s on four DIMMs, fixed
by 4600. Consumer non-ECC memory raises **no MCE** for a silent flip, and
@@ -763,7 +841,7 @@ linked runbooks.
`LimitMEMLOCK=infinity` (systemd is privileged; the value flows through
podman to runc without any raise). Upstream halogen's own run lines pass
NO `--ulimit` at all — copy vendor run lines before adding flags
(measured 2026-10-05 on the frank-large cutover).
(measured 2026-10-05 on the frank-dragoon cutover).
- **Kairic Edge** (if ever re-deployed): the pinned CK commit needs
`third_party/composable_kernel/include/ck/config.h` generated manually
(untracked; `git clean` removes it), and its greedy-argmax fastpath rejects
@@ -789,7 +867,7 @@ linked runbooks.
121 GiB box, pushed `available` to 3 GiB, and every later rate was suspect.
Record the child pid and kill it in a `finally`, then prove `free` came back
before believing a number. Kill by pid — never `pkill -f llama-server`,
which also hits frank-secondary.
which also hits frank-zealot.
**llama.cpp serving**
@@ -813,7 +891,7 @@ linked runbooks.
builds its context against whatever model the draft context holds. Two
context-backed types cannot share one process, and MTP's head only loads
when MTP **is** the draft model. **A LEGAL chain can still lose on a
recurrent model:** `draft-mtp,ngram-mod` on frank-large's qwen35 hybrid
recurrent model:** `draft-mtp,ngram-mod` on frank-dragoon's qwen35 hybrid
measured −15 % prose / −7 % block-copy (2026-10-03) — adding the ngram layer
logs `recurrent state rollback is not compatible with 'ngram-mod' -
disabling rollback support`, and MTP verification depends on that rollback,
@@ -855,7 +933,7 @@ linked runbooks.
free. That was a CONTIGUITY failure, not a total-bytes one, and no totals
model — the sweep's or the seam's — can see it.)
- **On the CUDA 3090 the same knob has NO interior optimum up to 2048, so
frank-secondary ships 4096/2048.** Measured 2026-10-05 by boot-pair A/B on the
frank-zealot ships 4096/2048.** Measured 2026-10-05 by boot-pair A/B on the
live service: at 8,928 prompt tokens ubatch 1024 gives 3,972 / 4,077 t/s and
ubatch 2048 gives 4,207 / 4,212 — **+4.6 %** — for **616 MiB** (free 2,849 →
2,279). Decode did not move (91.9–93.0 vs 93.8–94.3, i.e. noise), and a
@@ -864,9 +942,9 @@ linked runbooks.
than the ubatch cannot see it. The logical `--batch-size` was again free. And
the predecessor's "ubatch 2048 failed the scratch boot" was a shape
observation, not a law: the same file boots 2048 clean at 2 × 200,704. Bench
it by restarting the SERVICE through a transient `FRANK_SECONDARY_ENV_FILE`
it by restarting the SERVICE through a transient `FRANK_ZEALOT_ENV_FILE`
drop-in, never a scratch port — one card holds one server. See
[[frank-secondary-llama-mimo-q6k-2x200704]].
[[frank-zealot-llama-mimo-q6k-2x200704]].
- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same
size.** qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is
worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on a full-attention one
@@ -888,7 +966,7 @@ linked runbooks.
CTX_SIZE / PARALLEL`, and llama.cpp refuses a prompt longer than the SLOT
(`400 exceed_context_size_error: request (184001 tokens) exceeds the
available context size (180224 tokens)`) even when the pool has room. So
frank-secondary's 2 → 3 reslice moved each conversation from the model's native
frank-zealot's 2 → 3 reslice moved each conversation from the model's native
262,144 down to 180,224, and `models.yml` `contextWindow` had to follow or
the harness would discover the ceiling as a 400 on a long conversation. Side
effect worth knowing: the shorter slot made decode ~5 % FASTER (106.8 vs
@@ -899,9 +977,9 @@ linked runbooks.
day the freed space also bought the Q6_K rung (now 94.4 tok/s — the rung
itself is a further measured −10 %, weight-bandwidth, not slot geometry).
All moves measured; both slot directions cost or buy window, never pool.
See [[frank-secondary-llama-mimo-q6k-2x200704]].
See [[frank-zealot-llama-mimo-q6k-2x200704]].
- **Price a VRAM budget from two boots, never from a formula — and an mmproj
costs ~1.5x its file size.** The frank-secondary 3090 fit was solved from two
costs ~1.5x its file size.** The frank-zealot 3090 fit was solved from two
measurements: 2 × 262,144 with no projector = 20,190 MiB used (KV 12,800 ⇒
**non-KV 7,390**), 3 × 196,608 with the f16 projector = 22,972 used (KV
14,113 ⇒ **non-KV 8,859**). The difference is the vision tower: a 918 MB file
@@ -909,24 +987,24 @@ linked runbooks.
ride with it. Subtracting the two rows is exact; extrapolating weights+KV+
"a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7. Then
solve `KV ≤ card − non-KV − headroom` and round to a multiple of
`PARALLEL × 4096`. Recorded in `scripts/frank-secondary.env` so the next resize
`PARALLEL × 4096`. Recorded in `scripts/frank-zealot.env` so the next resize
starts from measurement.
- **`images` and `image_url` are NOT interchangeable: only the content part
reaches the model.** `--mmproj` loads, `usage.prompt_tokens` rises, and the
model still invents an answer if the client used the legacy top-level
`images` field on `/v1/chat/completions` — frank-secondary answered "Brown." for
`images` field on `/v1/chat/completions` — frank-zealot answered "Brown." for
a solid-red PNG that way and "Red." through the OpenAI `image_url` part. The
failure is silent because a hallucinated color is a plausible answer. Probe
vision with a solid color and expect an exact word.
- **`--jinja` is what `tools` needs.** On 2026-09-16 a subagent died with
`500 tools param requires --jinja flag`, from the ik_llama.cpp runtime that
then served frank-large: every tool-using agent on that endpoint was broken
then served frank-dragoon: every tool-using agent on that endpoint was broken
and the failure read like a model fault. Mainline llama.cpp 0.4.1 defaults
`--jinja` on and the serve script pins it; halogen also accepts `tools` and
returns `tool_calls`. Check tool support before routing a role at an endpoint.
- **Quantized KV on CUDA needs `GGML_CUDA_FA_ALL_QUANTS=ON`** — the same
option and the same ~12x prefill collapse described in the ROCm note above.
f16 KV needs nothing; frank-secondary ran f16 for that reason until
f16 KV needs nothing; frank-zealot ran f16 for that reason until
2026-09-17, when `build-cuda` was rebuilt with the option ON so the
endpoint could take q8_0 KV.
- **llama-bench separates a LIST with `/` and SWEEP values with `,`.** Both
@@ -944,7 +1022,7 @@ linked runbooks.
raises on anything else — HTTP 500, not a clamp — so the harness's levels
must be folded onto that set in `models.yml` (`compat.reasoningEffortMap`).
A role whose level is unmapped fails on its FIRST request, and the failure
names Jinja, not the model: every subagent on `frank-secondary` died with
names Jinja, not the model: every subagent on `frank-zealot` died with
`Unexpected reasoning effort high` until `high: xhigh` was added on
2026-09-16. Halogen accepts the wider set, so the two entries legitimately
differ.
@@ -964,7 +1042,7 @@ linked runbooks.
times (`RuntimeError: No HIP GPUs are available`).
- SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its
ports off the gluetun service.
- `amd_iommu=off` on Dankclaw (bought 13–16 % prefill for frank-large,
- `amd_iommu=off` on Dankclaw (bought 13–16 % prefill for frank-dragoon,
2026-09-11) permanently retired the XDNA2 NPU: the driver's SVA/PASID bind
needs an enabled IOMMU (`/dev/accel/accel0` fails `err=-19`). Revert path
in `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md`.
@@ -1018,9 +1096,9 @@ Active:
| Service | Runbook |
|---------|---------|
| frank-large (flash-server 0.16.2, abliterated Swift1.5 v2 .hgn + MTP always-on, 2×262,144, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md` |
| frank-secondary (llama.cpp CUDA, Qwen3.5-9B-Uncensored Q6_K, 2×200,704 exact f16 KV MTP-off, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md` |
| frank-primary (radiance StillDeadcode, Qwen3.8-Flash-Next-FP8 native, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md` |
| frank-dragoon (flash-server 0.16.2, abliterated Swift1.5 v2 .hgn + MTP always-on, 2×262,144, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-swift15.md` |
| frank-zealot (llama.cpp CUDA, Qwen3.5-9B-Uncensored Q6_K, 2×200,704 exact f16 KV MTP-off, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-zealot-llama-qwen35-9b-q6k-2x200704.md` |
| frank-archon (radiance StillDeadcode, Qwen3.8-Flash-Next-FP8 native, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-native-fp8.md` |
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
@@ -1028,41 +1106,41 @@ Retired (rollback targets and history):
| Runtime | Host | Window | Runbook |
|---------|------|--------|---------|
| llama.cpp CUDA Gemma4-26B-A4B QAT Q4_K_M, 1×200,704 f16-K/q8_0-V MTP-off (`frank-secondary`) | Dankclaw | 2026-10-07 → 10-07 (same-day swap; rollback target — payload on disk; owns the `attention_k_eq_v` pair-billing KV law and the gemma4-assistant drafter-cost boots) | `frank-secondary-llama-gemma4-q4km-200704.md` |
| ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-large-rocmfpx-q6agent-uncensored-mtp.md` |
| tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-primary`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-primary-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` |
| FreeToken PR #587 ROCm, Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 (`frank-primary`) | Dankserver | 2026-10-05 → 10-05 (rollback target — tag `frank-primary-freetoken-pr587-20261005`; trio + Dockerfile in the tag, image `localhost/frank-primary-freetoken:pr587-77a7201` and orcarouter payload on disk) | `frank-primary-freetoken-pr587.md` |
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K 2×200,704 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-03 → 10-05 (rollback target — payload on disk; owns the ubatch-2048 A/B; the Ornith swap of 10-05 was committed but never deployed, so this was the LIVE payload Swift replaced) | `frank-secondary-llama-mimo-q6k-2x200704.md` |
| llama.cpp Ornith-1.5-9B Q8_0, ONE 262,144 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-05 (committed 77decb5→380aafa, never deployed — superseded by Swift the same evening; payload on disk) | trail in `frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md` |
| llama.cpp CUDA Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2×114,688 draft-mtp q8_0 KV (`frank-secondary`) | Dankclaw | 2026-10-06 → 2026-10-07 (rollback target — payload on disk; owns the 3090 ubatch ladder A/B and the 34 KiB/token qwen35 KV law) | `frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md` |
| ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-large-rocmfpx-q8agent-mtp.md` |
| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-large-gufo-qwen38-27b.md` |
| vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` |
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-secondary`) | Dankclaw | 2026-10-01 → 10-02 (the rung returned ACTIVE 2026-10-03 at the 2×200,704 shape — `frank-secondary-llama-mimo-q6k-2x200704.md`) | `frank-secondary-llama-mimo-q6k.md` |
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q5_K_M 2×200,704 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-02 → 10-03 (rollback target — one `MODEL_FILE` line, buys back 875 MiB + the −10 % rung tax; owns the boot-pair table) | `frank-secondary-llama-mimo-q5km.md` |
| SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-secondary`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-secondary-sglang-mimo-v26.md` |
|tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-primary`)|Dankserver|2026-09-30 → 10-01|`frank-primary-glm53-flash-w4a16.md`|
| llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` |
| R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-primary`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-primary-r9v-radiance-qwen38-uncensored.md` |
| llama.cpp b11030 Qwen3.8-Flash-Next Q8_0 + shared MTP (`frank-primary`) | Dankserver | 2026-09-26 → 09-27 | `frank-primary-llamacpp-qwen38-flash-next-q8.md` |
| ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` |
| llama.cpp CUDA Gemma4-26B-A4B QAT Q4_K_M, 1×200,704 f16-K/q8_0-V MTP-off (`frank-zealot`) | Dankclaw | 2026-10-07 → 10-07 (same-day swap; rollback target — payload on disk; owns the `attention_k_eq_v` pair-billing KV law and the gemma4-assistant drafter-cost boots) | `frank-zealot-llama-gemma4-q4km-200704.md` |
| ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-dragoon`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-dragoon-rocmfpx-q6agent-uncensored-mtp.md` |
| tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-archon`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-archon-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` |
| FreeToken PR #587 ROCm, Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 (`frank-archon`) | Dankserver | 2026-10-05 → 10-05 (rollback target — tag `frank-archon-freetoken-pr587-20261005`; trio + Dockerfile in the tag, image `localhost/frank-archon-freetoken:pr587-77a7201` and orcarouter payload on disk) | `frank-archon-freetoken-pr587.md` |
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K 2×200,704 f16 KV (`frank-zealot`) | Dankclaw | 2026-10-03 → 10-05 (rollback target — payload on disk; owns the ubatch-2048 A/B; the Ornith swap of 10-05 was committed but never deployed, so this was the LIVE payload Swift replaced) | `frank-zealot-llama-mimo-q6k-2x200704.md` |
| llama.cpp Ornith-1.5-9B Q8_0, ONE 262,144 f16 KV (`frank-zealot`) | Dankclaw | 2026-10-05 (committed 77decb5→380aafa, never deployed — superseded by Swift the same evening; payload on disk) | trail in `frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688.md` |
| llama.cpp CUDA Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2×114,688 draft-mtp q8_0 KV (`frank-zealot`) | Dankclaw | 2026-10-06 → 2026-10-07 (rollback target — payload on disk; owns the 3090 ubatch ladder A/B and the 34 KiB/token qwen35 KV law) | `frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688.md` |
| ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-dragoon`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-dragoon-rocmfpx-q8agent-mtp.md` |
| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-dragoon`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-dragoon-gufo-qwen38-27b.md` |
| vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-dragoon`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-dragoon-vllm-mimo-v26-bf16.md` |
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-zealot`) | Dankclaw | 2026-10-01 → 10-02 (the rung returned ACTIVE 2026-10-03 at the 2×200,704 shape — `frank-zealot-llama-mimo-q6k-2x200704.md`) | `frank-zealot-llama-mimo-q6k.md` |
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q5_K_M 2×200,704 f16 KV (`frank-zealot`) | Dankclaw | 2026-10-02 → 10-03 (rollback target — one `MODEL_FILE` line, buys back 875 MiB + the −10 % rung tax; owns the boot-pair table) | `frank-zealot-llama-mimo-q5km.md` |
| SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-zealot`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-zealot-sglang-mimo-v26.md` |
|tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-archon`)|Dankserver|2026-09-30 → 10-01|`frank-archon-glm53-flash-w4a16.md`|
| llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-dragoon`) | Dankclaw | 2026-09-21 → 09-28 | `frank-dragoon-unsloth-qwen38-27b-q8xl.md` |
| R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-archon`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-archon-r9v-radiance-qwen38-uncensored.md` |
| llama.cpp b11030 Qwen3.8-Flash-Next Q8_0 + shared MTP (`frank-archon`) | Dankserver | 2026-09-26 → 09-27 | `frank-archon-llamacpp-qwen38-flash-next-q8.md` |
| ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-dragoon`) | Dankclaw | 2026-09-15 → 09-16 | `frank-dragoon-ik-uncensored.md` |
| vllm-radiance TP=2 MXFP4 + DFlash2 (`frank-medium`) | Dankserver | 2026-09-20 → 09-26 | `frank-medium-radiance-mxfp4.md` |
| Lucebox DS4.1 DeepSeek-V4.1-Flash ROCmFP23 (`frank-primary`) | Dankserver | 2026-09-26 (same-day swap) | `frank-primary-lucebox-ds4.md` |
| Lucebox DS4.1 DeepSeek-V4.1-Flash ROCmFP23 (`frank-archon`) | Dankserver | 2026-09-26 (same-day swap) | `frank-archon-lucebox-ds4.md` |
| kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` |
| halogen-flash-server Qwen3.8-Flash-Next w4b (`frank-large`) | Dankclaw | 2026-09-16 → 09-19 | `halogen-flash-server.md` |
| CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-large`) | Dankclaw | 2026-09-19 → 09-21 | `frank-large-ciru-orca.md` |
| unsloth llama.cpp b11030 Qwen3.8-Flash-Next UD-IQ4_XS (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-unsloth-qwen38-flash-next.md` |
| Kairic Edge Qwen3.8-27B IU4 W4A4 (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-kairic-edge-qwen38-27b.md` |
| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-secondary`) | Dankclaw | 2026-09-16 → 09-17 | `frank-secondary-qwen38-gsq-rco.md` |
| vLLM Gemma4-12B w4a16 (`frank-secondary`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-secondary-vllm-gemma4.md` |
| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-secondary`) | Dankclaw | 2026-09-17 → 10-01 | `frank-secondary-gemma4-12b.md` |
| halogen-flash-server Qwen3.8-Flash-Next w4b (`frank-dragoon`) | Dankclaw | 2026-09-16 → 09-19 | `halogen-flash-server.md` |
| CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-dragoon`) | Dankclaw | 2026-09-19 → 09-21 | `frank-dragoon-ciru-orca.md` |
| unsloth llama.cpp b11030 Qwen3.8-Flash-Next UD-IQ4_XS (`frank-dragoon`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-dragoon-unsloth-qwen38-flash-next.md` |
| Kairic Edge Qwen3.8-27B IU4 W4A4 (`frank-dragoon`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-dragoon-kairic-edge-qwen38-27b.md` |
| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-zealot`) | Dankclaw | 2026-09-16 → 09-17 | `frank-zealot-qwen38-gsq-rco.md` |
| vLLM Gemma4-12B w4a16 (`frank-zealot`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-zealot-vllm-gemma4.md` |
| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-zealot`) | Dankclaw | 2026-09-17 → 10-01 | `frank-zealot-gemma4-12b.md` |
| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
| eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` |
| Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` |
| CIRU ActiveFPX v3 / PromptForge | Dankclaw | through 2026-08-22 | `ciru-activefpx-promptforge.md` |
| DFlash2 Vulkan | Dankclaw | 2026-08-21 (same-day revert) | `dflash2-large-vulkan.md` |
| FastFlowLM NPU (`frank-npu`) | Dankclaw | twice, retired 2026-09-11 | `frank-npu.md`, `FastFlowLM NPU Inference.md` |
| FreeToken (official + Huihui FP8, `frank-secondary`) | Dankserver | 2026-09-02 → 09-15 | `freetoken-frank-small.md` |
| FreeToken (official + Huihui FP8, `frank-zealot`) | Dankserver | 2026-09-02 → 09-15 | `freetoken-frank-small.md` |
| FastMTP patched llama.cpp | Dankserver | retired 2026-08-18 | `qwen3.8-spec-matrix.md` |
| ik_llama.cpp Ornith CRACK | Dankserver | retired 2026-09-02 | `ikllama-ornith-dankserver.md` |
| llama.cpp layer split + DFlash2 (`frank-medium`) | Dankserver | 2026-09-16 → 09-20 | `frank-medium-llamacpp-dflash2.md` |
@@ -1086,7 +1164,7 @@ history.
| Unstaged local changes | **Commit or discard** — check `git status` before starting work; ask if unsure. |
| Deploying a committed change | **`scripts/deploy.sh <target>`** — pushes, pulls ff-only, plans, dispatches, health-checks. |
| Container without a Quadlet unit | **Give it one** — boot-persistent, self-healing `.container` via git. |
| Benchmarking | **Stop and mask the service first**, bench, unmask, restart, verify health. |
| Benchmarking | **Not part of a model swap** — swaps verify on the live service. A deliberate bench window: **stop and mask the service first**, bench, unmask, restart, verify health. |
| Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
| Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. |
+2 -2
View File
@@ -13,7 +13,7 @@ one deploy script.
| Host | User | IP | Podman stack | LLM service | LLM port |
|------|------|----|--------------|-------------|----------|
| dankserver | dank | 192.168.0.235 | media and library | frank-medium (ROCmFPX llama.cpp, Qwen3.8-27B AGENT) | 7770 |
| dankclaw | dank | 192.168.0.143 | AI, search, development, observability | frank-large (gufo, Qwen3.8-27B UD-Q8_K_XL) | 7770 |
| dankclaw | dank | 192.168.0.143 | AI, search, development, observability | frank-dragoon (gufo, Qwen3.8-27B UD-Q8_K_XL) | 7770 |
## Active Services
@@ -30,7 +30,7 @@ one deploy script.
| Gitea | dankclaw | gitea.dankclaw.local | 3001 |
| Grafana | dankclaw | grafana.dankclaw.local | 3002 |
| Prometheus | dankclaw | prometheus.dankclaw.local | 9090 |
| frank-large | dankclaw | — (bare endpoint) | 7770 |
| frank-dragoon | dankclaw | — (bare endpoint) | 7770 |
## Structure
+9 -9
View File
@@ -1,26 +1,26 @@
#!/usr/bin/env bash
# autoresearch.sh — canonical frank-primary benchmark entry point (dev box).
# autoresearch.sh — canonical frank-archon benchmark entry point (dev box).
#
# The service runs on dankserver (192.168.0.235:7770), the two R9700s live
# there, and the benchmark must own those cards, so the whole stop -> boot
# scratch -> bench -> restore sequence runs ON dankserver (see
# scripts/autoresearch-frank-primary.sh). This wrapper:
# 1. copies the candidate config (the repo's scripts/frank-primary.env, the
# scripts/autoresearch-frank-archon.sh). This wrapper:
# 1. copies the candidate config (the repo's scripts/frank-archon.env, the
# file the autoresearch loop edits) and the bench script to dankserver,
# 2. runs the bench over ssh, relaying its stdout (the METRIC lines) and
# exit code back here verbatim.
# It deliberately does NOT push to origin: the scp dirties dankserver's
# working tree, and the loop's uncommitted candidate config must not be
# pushed. Only this file and scripts/autoresearch-frank-primary.sh are
# pushed. Only this file and scripts/autoresearch-frank-archon.sh are
# committed; the candidate env rides over ssh each run.
set -uo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REMOTE=dankserver
REMOTE_REPO=/home/dank/serverAppConfigs
scp -q "$DIR/scripts/frank-primary.env" \
"$REMOTE:$REMOTE_REPO/scripts/frank-primary.env" || { echo "scp env failed" >&2; exit 1; }
scp -q "$DIR/scripts/autoresearch-frank-primary.sh" \
"$REMOTE:$REMOTE_REPO/scripts/autoresearch-frank-primary.sh" || { echo "scp script failed" >&2; exit 1; }
scp -q "$DIR/scripts/frank-archon.env" \
"$REMOTE:$REMOTE_REPO/scripts/frank-archon.env" || { echo "scp env failed" >&2; exit 1; }
scp -q "$DIR/scripts/autoresearch-frank-archon.sh" \
"$REMOTE:$REMOTE_REPO/scripts/autoresearch-frank-archon.sh" || { echo "scp script failed" >&2; exit 1; }
ssh "$REMOTE" "bash $REMOTE_REPO/scripts/autoresearch-frank-primary.sh $REMOTE_REPO/scripts/frank-primary.env"
ssh "$REMOTE" "bash $REMOTE_REPO/scripts/autoresearch-frank-archon.sh $REMOTE_REPO/scripts/frank-archon.env"
+1 -1
View File
File diff suppressed because one or more lines are too long
@@ -1,9 +1,9 @@
#!/usr/bin/env bash
# download-frank-primary-fp8.sh — fetch/verify the Qwen3.8-Flash-Next-FP8
# download-frank-archon-fp8.sh — fetch/verify the Qwen3.8-Flash-Next-FP8
# payload (131 safetensors shards, 172.78 GiB total, plus aux files) for the
# vLLM frank-primary deployment on dankserver.
# vLLM frank-archon deployment on dankserver.
#
# bash scripts/download-frank-primary-fp8.sh [dest_dir]
# bash scripts/download-frank-archon-fp8.sh [dest_dir]
#
# The manifest (scripts/manifests/qwen3.8-flash-next-fp8.manifest) carries
# per-file sizes + LFS sha256s from the Hub; hf-fetch.sh is idempotent — an
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
# Download the z-lab Qwen3.5-9B-DFlash drafter for frank-large's DFlash
# Download the z-lab Qwen3.5-9B-DFlash drafter for frank-dragoon's DFlash
# speculative-decode arm (vLLM --speculative-config method=dflash). Public
# repo — no auth header on the command line.
#
@@ -10,7 +10,7 @@
# the draft for that target.
#
# Usage:
# scripts/download-frank-large-dflash.sh [DEST_DIR] [--verify-only]
# scripts/download-frank-dragoon-dflash.sh [DEST_DIR] [--verify-only]
set -euo pipefail
REPO="z-lab/Qwen3.5-9B-DFlash"
@@ -1,14 +1,14 @@
#!/usr/bin/env bash
# Download the BF16 MiMo-V2.6-Distill-Qwen-9B payload for frank-large
# Download the BF16 MiMo-V2.6-Distill-Qwen-9B payload for frank-dragoon
# (vLLM/kyuz0 gfx1151, Strix Halo iGPU). Public repo — no auth header on the
# command line (a Bearer token in argv is world-readable via /proc).
#
# Per-file FRESH downloads, no resume: the house rule (see
# download-frank-secondary.sh); -f/--fail-with-body plus size+sha256 against the
# download-frank-zealot.sh); -f/--fail-with-body plus size+sha256 against the
# Hub LFS oid is what makes an access error fail instead of "succeeding".
#
# Usage:
# scripts/download-frank-large-mimo.sh [DEST_DIR] [--verify-only]
# scripts/download-frank-dragoon-mimo.sh [DEST_DIR] [--verify-only]
#
# The verified set below is exactly what the 2026-10-01 bench window served:
# 17.53 GiB of BF16 weights (4 shards, 18,819,720,848 B total with the rest).
@@ -1,14 +1,14 @@
#!/usr/bin/env bash
# Download the source Qwen3.5-9B checkpoint — the ONLY place this model
# family's MTP head (15 mtp.* tensors) exists. frank-large runs it as the
# family's MTP head (15 mtp.* tensors) exists. frank-dragoon runs it as the
# vLLM cross-model MTP draft (MiMo-V2.6-Distill-Qwen-9B itself ships zero
# mtp tensors; that is why SGLang's NEXTN on frank-secondary draws 0.00
# mtp tensors; that is why SGLang's NEXTN on frank-zealot draws 0.00
# acceptance). Public repo — no auth header on the command line.
#
# NOTE the odd upstream shard names (model.safetensors-0000N-of-00004...).
#
# Usage:
# scripts/download-frank-large-qwen35.sh [DEST_DIR] [--verify-only]
# scripts/download-frank-dragoon-qwen35.sh [DEST_DIR] [--verify-only]
set -euo pipefail
REPO="Qwen/Qwen3.5-9B"
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
# Download MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound for frank-secondary
# Download MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound for frank-zealot
# (SGLang, RTX 3090). Public repo — no auth header on the command line.
#
# Per-file FRESH downloads, no resume: `curl -C -` once produced a 213 MB
@@ -7,7 +7,7 @@
# the house rule.
#
# Usage:
# scripts/download-frank-secondary.sh [DEST_DIR] [--verify-only]
# scripts/download-frank-zealot.sh [DEST_DIR] [--verify-only]
#
# LFS shards are verified against the Hub LFS sha256; non-LFS text files
# against their declared byte size. Idempotent: complete files are skipped.
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
# bench-frank-large.sh — 64k-context benchmark for the frank-default-large stack.
# bench-frank-dragoon.sh — 64k-context benchmark for the frank-default-large stack.
#
# Measures generation throughput of Qwen3.6-27B Q8_K_P + DFlash + ngram-mod on
# a single backend at 64k context. It starts llama-server on a scratch port,
@@ -7,15 +7,15 @@
# prompt, and prints tokens/sec.
#
# Usage:
# ./scripts/bench-frank-large.sh <ROCm0|Vulkan0> [--generations N]
# ./scripts/bench-frank-dragoon.sh <ROCm0|Vulkan0> [--generations N]
#
# Example:
# ./scripts/bench-frank-large.sh ROCm0
# ./scripts/bench-frank-large.sh Vulkan0 --generations 3
# ./scripts/bench-frank-dragoon.sh ROCm0
# ./scripts/bench-frank-dragoon.sh Vulkan0 --generations 3
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
DEVICE="${1:?usage: bench-frank-large.sh <ROCm0|Vulkan0> [--generations N]}"
DEVICE="${1:?usage: bench-frank-dragoon.sh <ROCm0|Vulkan0> [--generations N]}"
GENERATIONS=1
if [ "${2:-}" = "--generations" ]; then
GENERATIONS="${3:?--generations needs a count}"
+1 -1
View File
@@ -41,7 +41,7 @@ SERVICE_HEALTH[dankclaw]="http://127.0.0.1:7770/health"
# Drop page caches before restart: with the model's pages fully cached, the
# KV/graph allocation stalls at load (VRAM stuck at 28 GiB). See AGENTS.md.
DROP_CACHES[dankclaw]=1
# Passwordless restart helper (root-owned, installed via /etc/sudoers.d/frank-large).
# Passwordless restart helper (root-owned, installed via /etc/sudoers.d/frank-dragoon).
# When present, deploy.sh uses it instead of inline sudo commands.
HELPER[dankclaw]="/usr/local/sbin/frank-restart-large.sh"
# Health wait in iterations of 2s. Cold loads after the cache drop measured
+2 -2
View File
@@ -1,6 +1,6 @@
# Tunables for the Ling-3.0-flash TRAIL on dankclaw (frank-large-ling).
# Tunables for the Ling-3.0-flash TRAIL on dankclaw (frank-dragoon-ling).
# This REPLACES frank-default-large (Qwen3.6-27B) temporarily. To switch
# back: systemctl stop frank-large-ling && systemctl start frank-default-large.
# back: systemctl stop frank-dragoon-ling && systemctl start frank-default-large.
# Runtime: llama.cpp FORK bailingmoe3-support; Vulkan + AMD GTT.
#
# Memory model (amdgpu Strix Halo gfx1151, 8x16GB LPDDR5 = 128 GiB physical):
@@ -1,5 +1,5 @@
[Unit]
Description=llama.cpp Ling-3.0-flash Q6_K Vulkan+MTP (frank-large-ling TRIAL)
Description=llama.cpp Ling-3.0-flash Q6_K Vulkan+MTP (frank-dragoon-ling TRIAL)
After=network.target
[Service]
+1 -1
View File
@@ -8,7 +8,7 @@
# Vulkan exposes as an ~111 GiB heap). MTP drafter is bundled in the same
# GGUF (block 42) - no separate --spec-draft-model needed.
#
# Tie-back: to restore the Qwen3.6-27B service, stop frank-large-ling and
# Tie-back: to restore the Qwen3.6-27B service, stop frank-dragoon-ling and
# start frank-default-large (its env/unit are committed and unchanged).
#
# Tunables come from frank-default-large.ling.env in this directory unless
@@ -4,11 +4,11 @@
# Runtime: llama.cpp llama-server (>= b10411).
#
# Tunables come from frank-default-large.env in this directory unless the
# FRANK_LARGE_ENV_FILE environment variable overrides that path.
# FRANK_DRAGOON_ENV_FILE environment variable overrides that path.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-default-large.env}"
ENV_FILE="${FRANK_DRAGOON_ENV_FILE:-$SCRIPT_DIR/frank-default-large.env}"
if [[ -f "$ENV_FILE" ]]; then
set -a
# shellcheck disable=SC1090
+1 -1
View File
File diff suppressed because one or more lines are too long
@@ -1,6 +1,6 @@
#!/usr/bin/env python3
"""
Decode throughput benchmark for frank-large.
Decode throughput benchmark for frank-dragoon.
Starts the server with a given --threads-batch value, sends an 8K+ token prompt,
generates 256 tokens, measures generation tok/s.
@@ -13,12 +13,12 @@ SERVE = "/home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server"
PORT = 7770
N_GEN = 256
PROMPT_FILE = "/tmp/bench-prompts/prompt_code.txt"
ENV_FILE = os.path.join(os.path.dirname(__file__), "frank-large.env")
ENV_FILE = os.path.join(os.path.dirname(__file__), "frank-dragoon.env")
SERVER_DIR = os.path.dirname(SERVE)
ROCM_PATH = "/opt/rocm"
def make_env():
"""Build the same env as serve-frank-large.sh."""
"""Build the same env as serve-frank-dragoon.sh."""
env = {**os.environ, "HSA_OVERRIDE_GFX_VERSION": "11.5.1", "HIP_VISIBLE_DEVICES": "0"}
# Source env file for tunables
@@ -59,7 +59,7 @@ def wait_ready():
def send_request(prompt, max_tokens=N_GEN):
"""Send chat completion, measure generation throughput."""
body = json.dumps({
"model": "frank-large",
"model": "frank-dragoon",
"messages": [{"role": "user", "content": prompt}],
"max_tokens": max_tokens,
"stream": False,
@@ -1,6 +1,6 @@
#!/usr/bin/env python3
"""
Benchmark n-gram speculative decoding vs no-spec for frank-large.
Benchmark n-gram speculative decoding vs no-spec for frank-dragoon.
Tests prompt-processing throughput with 16K token prompts (manageable request size).
Code-heavy prompt (high n-gram redundancy) vs prose prompt (low redundancy).
@@ -103,7 +103,7 @@ def start_server(port, spec_type):
def run_test(prompt_text, spec_type, port):
"""Send a request and extract prompt processing metrics."""
data = json.dumps({
"model": "frank-large",
"model": "frank-dragoon",
"messages": [{"role": "user", "content": prompt_text}],
"max_tokens": TEST_TOKENS,
"stream": False,
@@ -189,7 +189,7 @@ def main():
print(f" Prose: {prose_prompt_file.stat().st_size:,} bytes")
print("=" * 60)
print(" N-gram Mod Benchmark — frank-large")
print(" N-gram Mod Benchmark — frank-dragoon")
print(f" STRIX ROCmFP4 · 64K context · 16K prompts")
print("=" * 60)
@@ -1,19 +1,19 @@
#!/usr/bin/env bash
# Benchmark n-gram speculative decoding vs no-spec for frank-large.
# Benchmark n-gram speculative decoding vs no-spec for frank-dragoon.
#
# This tests prompt-processing throughput at 64k context with two distinct
# prompts: one code-heavy and one prose-heavy. Uses llama-server on scratch
# ports with the OpenAI-compatible API to measure prompt_eval_duration_ms.
#
# Run on Dankclaw after stopping the production service:
# sudo systemctl stop frank-large
# sudo systemctl stop frank-dragoon
# ./scripts/bench-ngram-mod.sh
# sudo systemctl start frank-large
# sudo systemctl start frank-dragoon
set -euo pipefail
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-large.env}"
ENV_FILE="${FRANK_DRAGOON_ENV_FILE:-$SCRIPT_DIR/frank-dragoon.env}"
if [[ -f "$ENV_FILE" ]]; then
set -a; source "$ENV_FILE"; set +a
fi
@@ -184,7 +184,7 @@ run_single_test() {
response=$(curl -s -X POST "http://127.0.0.1:$port/v1/completions" \
-H "Content-Type: application/json" \
-d "{
\"model\": \"frank-large\",
\"model\": \"frank-dragoon\",
\"prompt\": \"$prompt_text\",
\"max_tokens\": 128,
\"temperature\": 0,
@@ -220,7 +220,7 @@ run_single_test() {
}
echo "╔══════════════════════════════════════════════════════════╗"
echo "║ N-gram Mod Benchmark — frank-large ║"
echo "║ N-gram Mod Benchmark — frank-dragoon ║"
echo "║ STRIX ROCmFP4 · 64k context · two prompts ║"
echo "╚══════════════════════════════════════════════════════════╝"
echo ""
@@ -15,9 +15,9 @@ import time
import urllib.request
import urllib.error
SERVICE = "frank-large"
ENV_FILE = "/home/dank/serverAppConfigs/scripts/frank-large.env"
ALIAS = "frank-large"
SERVICE = "frank-dragoon"
ENV_FILE = "/home/dank/serverAppConfigs/scripts/frank-dragoon.env"
ALIAS = "frank-dragoon"
DEFAULT_HOST = "localhost"
DEFAULT_PORT = 7770
DEFAULT_MAX_TOKENS = 16384
@@ -1,4 +1,4 @@
# Tunables for frank-large (mradermacher i1-Q4_K_M of
# Tunables for frank-dragoon (mradermacher i1-Q4_K_M of
# orcarouter/Qwen3.8-Flash-Next-Uncensored on ik_llama.cpp, Dankclaw,
# RTX 3090 CUDA + Strix Halo CPU path, since 2026-09-15).
#
@@ -65,7 +65,7 @@ MMPROJ_PATH=/home/dank/models/mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K
CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/chat-template.jinja
# Alias + network.
ALIAS=frank-large
ALIAS=frank-dragoon
BIND_ADDR=0.0.0.0
LISTEN_PORT=7770
@@ -1,5 +1,5 @@
[Unit]
Description=ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M 177B MoE (dankclaw, frank-large)
Description=ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M 177B MoE (dankclaw, frank-dragoon)
After=network-online.target
Wants=network-online.target
@@ -12,8 +12,8 @@ Group=dank
# budget still needs room. Parity with the halogen quadlet's
# Ulimit=memlock=-1:-1.
LimitMEMLOCK=infinity
Environment="FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env"
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh
Environment="FRANK_DRAGOON_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-dragoon.env"
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-dragoon.sh
Restart=on-failure
RestartSec=10
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
# Serve mradermacher/Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M as frank-large
# Serve mradermacher/Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M as frank-dragoon
# via ik_llama.cpp on Dankclaw.
#
# Topology (ikawrakow's hybrid recipe; upstream issue #2402 for the PLE):
@@ -26,7 +26,7 @@
set -euo pipefail
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-large.env}"
ENV_FILE="${FRANK_DRAGOON_ENV_FILE:-$SCRIPT_DIR/frank-dragoon.env}"
if [[ -f "$ENV_FILE" ]]; then
set -a
# shellcheck disable=SC1090
@@ -66,7 +66,7 @@ fi
# CPU-overridden weights mmapped (src/llama-load-tensors.cpp); the env file
# sets it. Refuse to run without it.
if [[ "${GGML_CUDA_NO_PINNED_WEIGHTS:-}" != "1" ]]; then
echo "frank-large: refusing to start — GGML_CUDA_NO_PINNED_WEIGHTS=1 is mandatory on this host" >&2
echo "frank-dragoon: refusing to start — GGML_CUDA_NO_PINNED_WEIGHTS=1 is mandatory on this host" >&2
exit 1
fi
@@ -112,7 +112,7 @@ fi
: "${SPEC_DRAFT_POLL_BATCH:=1}"
# Text only: the server must not try to autoload the vision tower. Vision
# stays on frank-large (Dankclaw), which is where the fleet routes images.
# stays on frank-dragoon (Dankclaw), which is where the fleet routes images.
NO_MMPROJ="${NO_MMPROJ:-1}"
export HIP_VISIBLE_DEVICES
+1 -1
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env python3
"""autoresearch-bench.py — deterministic greedy battery for frank-large.
"""autoresearch-bench.py — deterministic greedy battery for frank-dragoon.
Runs ON Dankclaw (no LAN jitter) against the candidate container on :7770.
Fixed embedded corpus, temperature 0, streaming so decode speed is measured
@@ -1,12 +1,12 @@
#!/usr/bin/env bash
# autoresearch-frank-primary.sh — RUNS ON DANKSERVER, driven over ssh by the
# dev-box autoresearch.sh. It is the canonical frank-primary benchmark entry
# autoresearch-frank-archon.sh — RUNS ON DANKSERVER, driven over ssh by the
# dev-box autoresearch.sh. It is the canonical frank-archon benchmark entry
# point for the autoresearch loop.
#
# Method (mandated by the Benchmarking Policy): the production service serves
# real users and shares the two R9700s, so a bench beside it starves the GPUs
# and corrupts results. This script therefore:
# 1. stops the production frank-primary unit (root, relink symlink: stop,
# 1. stops the production frank-archon unit (root, relink symlink: stop,
# NOT mask — mask refuses to replace a symlinked unit file),
# 2. boots a scratch radiance container named `arbench` on the SAME two
# cards with the candidate engine argv (the env file the loop edits),
@@ -18,7 +18,7 @@
# prefill and decode rates for that single request.
# 5. prints METRIC lines and tears the scratch down,
# 6. restores production (systemctl start), via a trap so a failed bench
# never leaves frank-primary down (the stale-probe-tail lesson).
# never leaves frank-archon down (the stale-probe-tail lesson).
#
# Determinism: the prompt is built from a fixed, sorted corpus of repo files
# with a fixed random seed, so prompt_n is stable across runs. Sampling is
@@ -32,10 +32,10 @@
# /dev/dri, the .rad ro, the state dir rw, the tune cache mount. The ONLY
# differences: the container name (arbench, so prod's identity is untouched)
# and a FRESH prefix-cache disk tier (arbench/, wiped per run) so the cold
# prefill is cold — the prod tier at kvcache/frank-primary is left intact.
# prefill is cold — the prod tier at kvcache/frank-archon is left intact.
set -uo pipefail
ENV_FILE="${1:-/home/dank/serverAppConfigs/scripts/frank-primary.env}"
ENV_FILE="${1:-/home/dank/serverAppConfigs/scripts/frank-archon.env}"
SCRATCH=arbench
PORT=7770
HEALTH="http://127.0.0.1:$PORT/health"
@@ -45,7 +45,7 @@ LOG=/home/dank/logs/arbench.log
restore() {
podman rm -f "$SCRATCH" >/dev/null 2>&1 || true
sudo systemctl start frank-primary >/dev/null 2>&1 || true
sudo systemctl start frank-archon >/dev/null 2>&1 || true
}
trap restore EXIT
@@ -57,13 +57,13 @@ fi
set -a; . "$ENV_FILE"; set +a
# --- stop production; the scratch needs both cards to itself ----------------
sudo systemctl stop frank-primary >/dev/null 2>&1 || true
sudo systemctl stop frank-archon >/dev/null 2>&1 || true
podman rm -f "$SCRATCH" >/dev/null 2>&1 || true
# --- fresh prefix-cache tier for a cold prefill (prod's tier untouched) ------
FRESH_KV="$FRANK_PRIMARY_STATE_DIR/kvcache/$SCRATCH"
FRESH_KV="$FRANK_ARCHON_STATE_DIR/kvcache/$SCRATCH"
rm -rf "$FRESH_KV"; mkdir -p "$FRESH_KV"
mkdir -p "$FRANK_PRIMARY_STATE_DIR/tune"
mkdir -p "$FRANK_ARCHON_STATE_DIR/tune"
# --- boot the scratch: the production run line, scratch name + fresh KV ------
podman run -d --name "$SCRATCH" \
@@ -72,31 +72,31 @@ podman run -d --name "$SCRATCH" \
--security-opt seccomp=unconfined \
--ulimit memlock=-1:-1 \
--device /dev/kfd --device /dev/dri \
-v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
-v "$FRANK_PRIMARY_STATE_DIR":/data \
-v "$FRANK_PRIMARY_STATE_DIR/tune":/opt/radiance/share/radiance/tune \
-v "$FRANK_PRIMARY_TEMPLATE":/templates/template.jinja:ro \
"$FRANK_PRIMARY_IMAGE" \
--model "/models/$FRANK_PRIMARY_MODEL_FILE" \
-v "$FRANK_ARCHON_MODEL_DIR":/models:ro \
-v "$FRANK_ARCHON_STATE_DIR":/data \
-v "$FRANK_ARCHON_STATE_DIR/tune":/opt/radiance/share/radiance/tune \
-v "$FRANK_ARCHON_TEMPLATE":/templates/template.jinja:ro \
"$FRANK_ARCHON_IMAGE" \
--model "/models/$FRANK_ARCHON_MODEL_FILE" \
--host :: --port "$PORT" \
--served-model-name "$FRANK_PRIMARY_SERVED_NAME" \
--served-model-name "$FRANK_ARCHON_SERVED_NAME" \
--override-chat-template /templates/template.jinja \
--tp "$FRANK_PRIMARY_TP" --tp-wire "$FRANK_PRIMARY_TP_WIRE" \
--max-num-seqs "$FRANK_PRIMARY_MAX_SEQS" \
--max-model-len "$FRANK_PRIMARY_MAX_MODEL_LEN" \
--placement "$FRANK_PRIMARY_PLACEMENT" \
--host-pool-mib "$FRANK_PRIMARY_HOST_POOL_MIB" \
--gpu-headroom-mib "$FRANK_PRIMARY_GPU_HEADROOM_MIB" \
--vram-weights-mib "$FRANK_PRIMARY_VRAM_WEIGHTS_MIB" \
--vram-kv-mib "$FRANK_PRIMARY_VRAM_KV_MIB" \
--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
--prefix-cache-host-mib "$FRANK_PRIMARY_PREFIX_HOST_MIB" \
--tp "$FRANK_ARCHON_TP" --tp-wire "$FRANK_ARCHON_TP_WIRE" \
--max-num-seqs "$FRANK_ARCHON_MAX_SEQS" \
--max-model-len "$FRANK_ARCHON_MAX_MODEL_LEN" \
--placement "$FRANK_ARCHON_PLACEMENT" \
--host-pool-mib "$FRANK_ARCHON_HOST_POOL_MIB" \
--gpu-headroom-mib "$FRANK_ARCHON_GPU_HEADROOM_MIB" \
--vram-weights-mib "$FRANK_ARCHON_VRAM_WEIGHTS_MIB" \
--vram-kv-mib "$FRANK_ARCHON_VRAM_KV_MIB" \
--kv-cache-dtype "$FRANK_ARCHON_KV_DTYPE" \
--prefix-cache-host-mib "$FRANK_ARCHON_PREFIX_HOST_MIB" \
--prefix-cache-dir "/data/kvcache/$SCRATCH" \
--prefix-cache-disk-mib "$FRANK_PRIMARY_PREFIX_DISK_MIB" \
--ngram-placement "$FRANK_PRIMARY_NGRAM_PLACEMENT" \
--num-speculative-tokens "$FRANK_PRIMARY_SPEC_TOKENS" \
--max-num-batched-tokens "$FRANK_PRIMARY_MAX_BATCHED_TOKENS" \
${FRANK_PRIMARY_EXTRA_ARGS:+$FRANK_PRIMARY_EXTRA_ARGS} \
--prefix-cache-disk-mib "$FRANK_ARCHON_PREFIX_DISK_MIB" \
--ngram-placement "$FRANK_ARCHON_NGRAM_PLACEMENT" \
--num-speculative-tokens "$FRANK_ARCHON_SPEC_TOKENS" \
--max-num-batched-tokens "$FRANK_ARCHON_MAX_BATCHED_TOKENS" \
${FRANK_ARCHON_EXTRA_ARGS:+$FRANK_ARCHON_EXTRA_ARGS} \
> "$LOG" 2>&1
if [ $? -ne 0 ]; then echo "scratch boot failed; see $LOG" >&2; exit 1; fi
@@ -1,19 +1,19 @@
#!/usr/bin/env bash
# frank-primary-arms.sh — run a SEQUENCE of frank-primary bench arms and keep
# frank-archon-arms.sh — run a SEQUENCE of frank-archon bench arms and keep
# every arm's evidence. Runs on the operator box; each arm is one full window
# of ./autoresearch.sh (stop+mask -> boot the arm as the service -> measure ->
# restore production and verify), so the fleet is only ever down for one arm at
# a time and the production unit is back up between arms.
#
# An arm is the repo's own frank-primary.env plus an OVERLAY of engine knobs
# An arm is the repo's own frank-archon.env plus an OVERLAY of engine knobs
# appended after it (later assignment wins when the serve script sources it),
# staged through FRANK_PRIMARY_AR_ENV_FILE so the checkout is never mutated and
# staged through FRANK_ARCHON_AR_ENV_FILE so the checkout is never mutated and
# each arm gets its own fingerprint (the memo can never answer one arm with
# another arm's numbers).
#
# Usage: scripts/autoresearch/frank-primary-arms.sh [arm ...]
# scripts/autoresearch/frank-primary-arms.sh --list
# Output, under /tmp/frank-primary-arms-<UTC stamp>/ (override with ARMS_DIR):
# Usage: scripts/autoresearch/frank-archon-arms.sh [arm ...]
# scripts/autoresearch/frank-archon-arms.sh --list
# Output, under /tmp/frank-archon-arms-<UTC stamp>/ (override with ARMS_DIR):
#
# <arm>.out full autoresearch output (metrics + engine evidence)
# <arm>.env the exact staged env
@@ -26,8 +26,8 @@ set -uo pipefail
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
REPO="$(cd -- "$SCRIPT_DIR/../.." && pwd)"
BASE_ENV="$REPO/scripts/frank-primary.env"
ARMS_DIR="${ARMS_DIR:-/tmp/frank-primary-arms-$(date -u +%Y%m%dT%H%M%SZ)}"
BASE_ENV="$REPO/scripts/frank-archon.env"
ARMS_DIR="${ARMS_DIR:-/tmp/frank-archon-arms-$(date -u +%Y%m%dT%H%M%SZ)}"
# --- arms ------------------------------------------------------------------
# baseline: the shipped shape E, measured with THIS harness so the comparison is
@@ -45,18 +45,18 @@ LADDER3='{"cudagraph_capture_sizes":[4,8,12],"max_cudagraph_capture_size":12,"in
# the engine as {temperature:0.7}.
GEN07='{"temperature":0.7}'
OVER_B=(
"FRANK_PRIMARY_SPEC='$SPEC3'"
"FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
"FRANK_ARCHON_SPEC='$SPEC3'"
"FRANK_ARCHON_COMPCONFIG='$LADDER3'"
'FRANK_ARCHON_GEN_CONFIG='"'$GEN07'"
)
# B2: identical to B with the default (7.60 GiB) overhead, i.e. precise memory
# OFF. It is the fallback if B dies at graph capture, and it prices the arena
# that precise memory buys (7,522 slots -> ~6,496).
OVER_B2=(
"FRANK_PRIMARY_SPEC='$SPEC3'"
"FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
'FRANK_PRIMARY_PRECISE_MEMORY='
"FRANK_ARCHON_SPEC='$SPEC3'"
"FRANK_ARCHON_COMPCONFIG='$LADDER3'"
'FRANK_ARCHON_GEN_CONFIG='"'$GEN07'"
'FRANK_ARCHON_PRECISE_MEMORY='
)
# C: the vendor's concurrency shape — SEQS 16 with the [4..32] ladder and util
# 0.95. This is also the direct test of the theory that pinning the capture
@@ -64,12 +64,12 @@ OVER_B2=(
# the constant 3, which is NOT in the default ladder; 4 is).
LADDER16='{"cudagraph_capture_sizes":[4,8,12,16,20,24,28,32],"max_cudagraph_capture_size":32,"inductor_compile_config":{"combo_kernels":false,"benchmark_combo_kernel":false}}'
OVER_C=(
"FRANK_PRIMARY_SPEC='$SPEC3'"
"FRANK_PRIMARY_COMPCONFIG='$LADDER16'"
'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
'FRANK_PRIMARY_SEQS=16'
'FRANK_PRIMARY_MNBT=4096'
'FRANK_PRIMARY_UTIL=0.95'
"FRANK_ARCHON_SPEC='$SPEC3'"
"FRANK_ARCHON_COMPCONFIG='$LADDER16'"
'FRANK_ARCHON_GEN_CONFIG='"'$GEN07'"
'FRANK_ARCHON_SEQS=16'
'FRANK_ARCHON_MNBT=4096'
'FRANK_ARCHON_UTIL=0.95'
)
# D/E: the CONTEXT arms. Measured 2026-10-02 (arms run 20261002T233855Z): the
@@ -83,11 +83,11 @@ OVER_C=(
# D is arm B with EXACTLY ONE variable changed (the KV bytes), so its result
# says whether the 4k-depth collapse (B 23.09 vs baseline 33.99) was KV or the
# batch shape.
KV401K='FRANK_PRIMARY_KV_BYTES=4050000000'
KV401K='FRANK_ARCHON_KV_BYTES=4050000000'
OVER_D=(
"FRANK_PRIMARY_SPEC='$SPEC3'"
"FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
"FRANK_ARCHON_SPEC='$SPEC3'"
"FRANK_ARCHON_COMPCONFIG='$LADDER3'"
'FRANK_ARCHON_GEN_CONFIG='"'$GEN07'"
"$KV401K"
)
# E: D with C's max_num_batched_tokens (4096 — C is the only arm that did NOT
@@ -97,12 +97,12 @@ OVER_D=(
# only 0.32 GiB above the allowance while the CUDA-graph cost is an ANALYTIC
# estimate (C's was 1.47 GiB over 24 graphs), so it is measured, not assumed.
OVER_E=(
"FRANK_PRIMARY_SPEC='$SPEC3'"
"FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
"FRANK_ARCHON_SPEC='$SPEC3'"
"FRANK_ARCHON_COMPCONFIG='$LADDER3'"
'FRANK_ARCHON_GEN_CONFIG='"'$GEN07'"
"$KV401K"
'FRANK_PRIMARY_MNBT=4096'
'FRANK_PRIMARY_UTIL=0.99'
'FRANK_ARCHON_MNBT=4096'
'FRANK_ARCHON_UTIL=0.99'
)
arm_overlay() {
@@ -142,7 +142,7 @@ for arm in "${ARMS[@]}"; do
# holds JSON must still be one valid JSON object after the file is SOURCED
# (that is the step that silently eats unquoted braces).
if ! bash -c 'set -a; . "$1" >/dev/null; set +a
for k in FRANK_PRIMARY_SPEC FRANK_PRIMARY_COMPCONFIG FRANK_PRIMARY_GEN_CONFIG; do
for k in FRANK_ARCHON_SPEC FRANK_ARCHON_COMPCONFIG FRANK_ARCHON_GEN_CONFIG; do
v="${!k:-}"
[ -n "$v" ] || continue
case "$v" in
@@ -158,7 +158,7 @@ for arm in "${ARMS[@]}"; do
fi
echo "[arms] === $arm: opening window $(date -u +%H:%M:%S)Z ===" >&2
FRANK_PRIMARY_AR_ENV_FILE="$ARMS_DIR/$arm.env" \
FRANK_ARCHON_AR_ENV_FILE="$ARMS_DIR/$arm.env" \
"$REPO/autoresearch.sh" >"$ARMS_DIR/$arm.out" 2>&1
rc=$?
echo "$rc" >"$ARMS_DIR/$arm.rc"
@@ -1,5 +1,5 @@
#!/usr/bin/env python3
"""frank-primary decode measurement — runs INSIDE the engine container.
"""frank-archon decode measurement — runs INSIDE the engine container.
The method is the fleet's established one (the 2026-09-29 / 2026-10-01
tuning campaigns, /home/dank/mtp_measure.py): SSE chunk counting is invalid
@@ -28,7 +28,7 @@ Workload (fixed, deterministic, offline):
Env:
AR_BASE endpoint (default http://127.0.0.1:7770)
AR_MODEL served name (default frank-primary)
AR_MODEL served name (default frank-archon)
AR_PROMPT_CEILING hard prompt ceiling in tokens (default 100000)
AR_TEMPS comma-separated sampling temperatures (default "0,0.7")
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
# frank-primary-window.sh — one full bench window for frank-primary, run ON
# frank-archon-window.sh — one full bench window for frank-archon, run ON
# DANKSERVER as the deploy user (invoked over ssh by ./autoresearch.sh).
#
# Discipline (AGENTS.md Benchmarking Policy): the production unit is
@@ -12,9 +12,9 @@
#
# Staging layout (created by the orchestrator, ephemeral — never a config
# source; the repo checkout is):
# /home/dank/.autoresearch-frank-primary/
# /home/dank/.autoresearch-frank-archon/
# serve.sh staged candidate serve script
# frank-primary.env staged candidate env (the candidate knobs)
# frank-archon.env staged candidate env (the candidate knobs)
# measure.py the in-container measure client
# hash candidate fingerprint
# run.json result (metrics + boot + kv + fingerprint)
@@ -27,16 +27,16 @@
# outcome; a failed restore forces exit 1).
set -uo pipefail
STAGE="${FRANK_PRIMARY_AR_STAGE:-/home/dank/.autoresearch-frank-primary}"
UNIT="${FRANK_PRIMARY_AR_UNIT:-frank-primary}"
CTR="${FRANK_PRIMARY_AR_CTR:-frank-primary-qwen}"
STAGE="${FRANK_ARCHON_AR_STAGE:-/home/dank/.autoresearch-frank-archon}"
UNIT="${FRANK_ARCHON_AR_UNIT:-frank-archon}"
CTR="${FRANK_ARCHON_AR_CTR:-frank-archon-qwen}"
HEALTH_URL="http://127.0.0.1:7770/health"
MODELS_URL="http://127.0.0.1:7770/v1/models"
BOOT_TIMEOUT="${FRANK_PRIMARY_AR_BOOT_TIMEOUT:-1200}"
RESTORE_TIMEOUT="${FRANK_PRIMARY_AR_RESTORE_TIMEOUT:-1200}"
MEASURE_TIMEOUT="${FRANK_PRIMARY_AR_MEASURE_TIMEOUT:-1800}"
SERVED="$(sed -n 's/^FRANK_PRIMARY_SERVED_NAME=//p' "$STAGE/frank-primary.env" 2>/dev/null | head -1)"
SERVED="${SERVED:-frank-primary}"
BOOT_TIMEOUT="${FRANK_ARCHON_AR_BOOT_TIMEOUT:-1200}"
RESTORE_TIMEOUT="${FRANK_ARCHON_AR_RESTORE_TIMEOUT:-1200}"
MEASURE_TIMEOUT="${FRANK_ARCHON_AR_MEASURE_TIMEOUT:-1800}"
SERVED="$(sed -n 's/^FRANK_ARCHON_SERVED_NAME=//p' "$STAGE/frank-archon.env" 2>/dev/null | head -1)"
SERVED="${SERVED:-frank-archon}"
RESTORED=0
@@ -72,7 +72,7 @@ capture_bench_failure() {
echo "=== staged serve.out (tail 60) ==="
tail -60 "$STAGE/serve.out" 2>/dev/null || true
echo "=== engine log (tail 60) ==="
tail -60 /home/dank/logs/frank_primary.log 2>/dev/null || true
tail -60 /home/dank/logs/frank_archon.log 2>/dev/null || true
} >"$STAGE/fail.log" 2>&1 || true
grep -E 'OutOfMemory|Tried to allocate|Engine core initialization failed|Error|error' \
"$STAGE/fail.log" 2>/dev/null | tail -25 >&2 || true
@@ -114,7 +114,7 @@ trap 'restore; rc=$?; [ "$rc" -ne 0 ] && exit 1' EXIT
main() {
local f
for f in serve.sh frank-primary.env measure.py hash; do
for f in serve.sh frank-archon.env measure.py hash; do
[ -f "$STAGE/$f" ] || { log "staged file missing: $STAGE/$f"; exit 1; }
done
mkdir -p "$STAGE/last"
@@ -133,7 +133,7 @@ main() {
t0="$(date +%s)"
log "starting bench instance from staged config (fingerprint $(tr -d '[:space:]' <"$STAGE/hash"))"
FRANK_PRIMARY_ENV_FILE="$STAGE/frank-primary.env" \
FRANK_ARCHON_ENV_FILE="$STAGE/frank-archon.env" \
setsid nohup bash "$STAGE/serve.sh" >"$STAGE/serve.out" 2>&1 </dev/null &
echo "$!" >"$STAGE/serve.pid"
+2 -2
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
# Decode throughput benchmark for frank-large.
# Decode throughput benchmark for frank-dragoon.
# Measures generation tok/s with 8K+ token prompts and 256 generated tokens.
set -euo pipefail
@@ -75,7 +75,7 @@ print(json.dumps(' '.join(words)))
t0=$(date +%s%N)
resp=$(curl -s -m 600 http://127.0.0.1:$PORT/v1/chat/completions \
-H "Content-Type: application/json" \
-d "{\"model\":\"frank-large\",\"messages\":[{\"role\":\"user\",\"content\":$prompt}],\"max_tokens\":$N_GEN,\"stream\":false}" 2>&1)
-d "{\"model\":\"frank-dragoon\",\"messages\":[{\"role\":\"user\",\"content\":$prompt}],\"max_tokens\":$N_GEN,\"stream\":false}" 2>&1)
t1=$(date +%s%N)
if ! echo "$resp" | python3 -c "import sys,json; json.load(sys.stdin)" 2>/dev/null; then
+1 -1
View File
@@ -1,6 +1,6 @@
#!/usr/bin/env python3
"""
HTTP-based prefill-throughput sweep for frank-large.
HTTP-based prefill-throughput sweep for frank-dragoon.
Starts the server with a given --threads-batch value, sends a 64K-prompt via
the chat API (max_tokens=0), measures prompt_tokens/wall_time.
+2 -2
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
# Threads-batch sweep for frank-large on dankclaw
# Threads-batch sweep for frank-dragoon on dankclaw
# Benches prefill throughput at 65536 prompt tokens across threads-batch values.
set -uo pipefail
@@ -10,7 +10,7 @@ TB_LIST=(16 32 48 64 80 96 128)
ENV_FILE="/home/dank/GitPackages/ROCmFPX/build-qwen38/bin"
echo "========================================"
echo "THREADS-BATCH SWEEP — frank-large"
echo "THREADS-BATCH SWEEP — frank-dragoon"
echo "Model: $(basename "$MODEL") ($(du -h "$MODEL" | cut -f1))"
echo "Threads-batch values: ${TB_LIST[*]}"
echo "Prompt: 65536 tokens, n-gen: 0 (prefill-only)"
+4 -4
View File
@@ -1,14 +1,14 @@
#!/usr/bin/env bash
# Benchmark ubatch with llama-bench for frank-large.
# Benchmark ubatch with llama-bench for frank-dragoon.
#
# This is intentionally a ubatch-only benchmark. It uses one llama-bench
# process, an 8K prompt workload, and equal batch/ubatch values. No server
# process, HTTP requests, or log scraping are used.
#
# Run on Dankclaw after stopping frank-large:
# sudo systemctl stop frank-large
# Run on Dankclaw after stopping frank-dragoon:
# sudo systemctl stop frank-dragoon
# ./scripts/bench-ubatch.sh
# sudo systemctl start frank-large
# sudo systemctl start frank-dragoon
set -euo pipefail
: "${LLAMA_BENCH:=/home/dank/GitPackages/ROCmFPX/build-strix-rocmfp4/bin/llama-bench}"
@@ -1,11 +1,11 @@
#!/usr/bin/env bash
# build-frank-primary-rad.sh — build frank-primary's .rad container from the
# build-frank-archon-rad.sh — build frank-archon's .rad container from the
# OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint, at NATIVE precision.
#
# bash scripts/build-frank-primary-rad.sh # convert
# bash scripts/build-frank-primary-rad.sh --plan-only # declare, no I/O
# bash scripts/build-frank-primary-rad.sh --restage # redo staging
# bash scripts/build-frank-primary-rad.sh --force # rebuild existing
# bash scripts/build-frank-archon-rad.sh # convert
# bash scripts/build-frank-archon-rad.sh --plan-only # declare, no I/O
# bash scripts/build-frank-archon-rad.sh --restage # redo staging
# bash scripts/build-frank-archon-rad.sh --force # rebuild existing
#
# WHY A STAGING DIR (the one non-obvious step): rad-convert selects its
# architecture plugin by the checkpoint's quantisation DESCRIPTOR —
@@ -55,8 +55,8 @@ OUT="$OUT_DIR/qwen4exp-fp8-native.rad"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck disable=SC1091
source "$SCRIPT_DIR/frank-primary.env"
IMAGE="$FRANK_PRIMARY_IMAGE"
source "$SCRIPT_DIR/frank-archon.env"
IMAGE="$FRANK_ARCHON_IMAGE"
plan_only=0 restage=0 force=0
for a in "$@"; do
@@ -69,11 +69,11 @@ for a in "$@"; do
done
if [ ! -f "$SRC/config.json" ]; then
echo "ERROR: raw checkpoint missing: $SRC (run scripts/download-frank-primary.sh --fetch)" >&2
echo "ERROR: raw checkpoint missing: $SRC (run scripts/download-frank-archon.sh --fetch)" >&2
exit 1
fi
echo "verifying raw checkpoint first..."
bash "$SCRIPT_DIR/download-frank-primary.sh"
bash "$SCRIPT_DIR/download-frank-archon.sh"
if [ -f "$OUT" ] && [ "$plan_only" -eq 0 ] && [ "$force" -eq 0 ]; then
echo "container already exists: $OUT"
@@ -83,7 +83,7 @@ fi
# ---- staging ----------------------------------------------------------------
if [ "$restage" -eq 1 ] && [ -e "$STAGE" ]; then
TRASH="$HOME/.AGENTTRASH/$(date +%Y%m%d)-frank-primary-stage"
TRASH="$HOME/.AGENTTRASH/$(date +%Y%m%d)-frank-archon-stage"
mkdir -p "$TRASH"
mv "$STAGE" "$TRASH/stage-$(date +%H%M%S)"
echo "old staging dir moved to $TRASH"
@@ -138,7 +138,7 @@ CONVERT_ARGS=(/models/stage-qwen4exp-fp8 -o /models/rad/qwen4exp-fp8-native.rad
# exactly and the plugin's one-element scale plane is produced.
CONVERT_ARGS+=(--quant "blk.*.ple_ngram.weight=rtn:codes=fp8_e4m3,block=*x*,scale=bf16,rule=fixed,scale_value=$PLE_SCALE")
[ "$force" -eq 1 ] && [ -f "$OUT" ] && {
TRASH="$HOME/.AGENTTRASH/$(date +%Y%m%d)-frank-primary-rad"
TRASH="$HOME/.AGENTTRASH/$(date +%Y%m%d)-frank-archon-rad"
mkdir -p "$TRASH"
mv "$OUT" "$TRASH/qwen4exp-fp8-native.rad.$(date +%H%M%S)"
echo "previous container moved to $TRASH"
@@ -158,8 +158,8 @@ echo "rad-info summary:"
podman run --rm --entrypoint rad-info -v /home/dank/models:/models:ro \
"$IMAGE" "/models/rad/qwen4exp-fp8-native.rad" | head -20
MIB=$(( $(stat -c %s "$OUT") / 1048576 ))
echo "container: $OUT (${MIB} MiB; env floor $FRANK_PRIMARY_RAD_MIN_MIB MiB)"
if [ "$MIB" -lt "$FRANK_PRIMARY_RAD_MIN_MIB" ]; then
echo "container: $OUT (${MIB} MiB; env floor $FRANK_ARCHON_RAD_MIN_MIB MiB)"
if [ "$MIB" -lt "$FRANK_ARCHON_RAD_MIN_MIB" ]; then
echo "ERROR: container below the env floor — the convert was truncated" >&2
exit 1
fi
+2 -2
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
# Build the gufo runtime that serves frank-large on Dankclaw.
# Build the gufo runtime that serves frank-dragoon on Dankclaw.
#
# bash scripts/build-gufo.sh # build + install the pinned tag
# bash scripts/build-gufo.sh --verify # only check the installed tree
@@ -93,4 +93,4 @@ cmake --install build/release
echo "== verify =="
"$PREFIX/bin/gufo" diagnose
echo "frank-large runtime ready: $PREFIX/bin/gufo ($GUFO_TAG @ $commit)"
echo "frank-dragoon runtime ready: $PREFIX/bin/gufo ($GUFO_TAG @ $commit)"
+4 -4
View File
@@ -1,11 +1,11 @@
#!/usr/bin/env bash
# Build the ROCmFPX runtime that serves frank-large on Dankclaw.
# Build the ROCmFPX runtime that serves frank-dragoon on Dankclaw.
#
# bash scripts/build-rocmfpx.sh # clone/fetch + build the pin
# bash scripts/build-rocmfpx.sh --verify # only check the installed tree
#
# Runtime: kingjones30/ROCmFPX — charlie12345/ROCmFPX plus the model-card
# patches, i.e. the fork that owns the ROCmFPx tensor formats. frank-large's
# patches, i.e. the fork that owns the ROCmFPx tensor formats. frank-dragoon's
# payload is ftype 115 (`Q8_0_ROCMFPX_AGENT`): 311 tensors of
# GGML_TYPE_Q8_0_ROCMFPX (103), a type that exists ONLY in this fork, so
# stock llama.cpp, gufo and every other loader on the host reject the file.
@@ -30,7 +30,7 @@
set -euo pipefail
ROCMFPX_REPO="${ROCMFPX_REPO:-https://github.com/kingjones30/ROCmFPX.git}"
# The commit frank-large was qualified with (measured 2026-10-02).
# The commit frank-dragoon was qualified with (measured 2026-10-02).
ROCMFPX_COMMIT="${ROCMFPX_COMMIT:-36e9acd40e10a87cd3c3ef8ec734668757dc8520}"
SRC_DIR="${ROCMFPX_SRC_DIR:-/home/dank/GitPackages/kingjones-ROCmFPX}"
BUILD_DIR="${ROCMFPX_BUILD_DIR:-$SRC_DIR/build}"
@@ -86,4 +86,4 @@ cmake --build "$BUILD_DIR" --target llama-server llama-quantize -j "$JOBS"
echo "== verify =="
"$BUILD_DIR/bin/llama-server" --version | head -1
echo "frank-large runtime ready: $BUILD_DIR/bin/llama-server ($ROCMFPX_COMMIT)"
echo "frank-dragoon runtime ready: $BUILD_DIR/bin/llama-server ($ROCMFPX_COMMIT)"
+3 -3
View File
@@ -57,9 +57,9 @@ else
echo " none"
fi
section "frank-primary unit"
printf ' active: %s\n' "$(systemctl is-active frank-primary 2>/dev/null | head -1)"
printf ' enabled: %s\n' "$(systemctl is-enabled frank-primary 2>/dev/null | head -1)"
section "frank-archon unit"
printf ' active: %s\n' "$(systemctl is-active frank-archon 2>/dev/null | head -1)"
printf ' enabled: %s\n' "$(systemctl is-enabled frank-archon 2>/dev/null | head -1)"
section "glm serve tail"
if podman ps -a --format '{{.Names}}' 2>/dev/null | grep -qx "$GLM_CTR"; then
+23 -23
View File
@@ -61,9 +61,9 @@ SERVICE_ORDER = (
"sonarr",
"lidarr",
"readarr",
"frank-large",
"frank-secondary",
"frank-primary",
"frank-dragoon",
"frank-zealot",
"frank-archon",
)
DANKCLAW_SERVICES = {
@@ -93,8 +93,8 @@ DANKCLAW_SERVICES = {
"authentik-outpost-ldap",
"caddy",
"homepage",
"frank-large",
"frank-secondary",
"frank-dragoon",
"frank-zealot",
}
DANKSERVER_SERVICES = {
@@ -113,7 +113,7 @@ DANKSERVER_SERVICES = {
"sonarr",
"lidarr",
"readarr",
"frank-primary",
"frank-archon",
"node-exporter",
}
@@ -316,7 +316,7 @@ def services_for_path(path: str, target: str) -> set[str]:
if len(parts) >= 2 and parts[1] == "open-webui":
_add(services, "open-webui")
return services
# frank-large left the quadlet shape on 2026-09-19 (CIRU v4.4.1 host
# frank-dragoon left the quadlet shape on 2026-09-19 (CIRU v4.4.1 host
# build, root unit): its files live in scripts/ and map there.
if len(parts) >= 2 and parts[1] == "observability":
# The five observability containers became Quadlet units on
@@ -371,10 +371,10 @@ def services_for_path(path: str, target: str) -> set[str]:
if name == "node-exporter":
_add(services, "node-exporter")
return services
# frank-primary's quadlet-era directory left the tree on 2026-09-26
# frank-archon's quadlet-era directory left the tree on 2026-09-26
# (the Lucebox DS4.1 unit was retired for the llama.cpp root unit —
# the trio in scripts/ now maps it). Nothing lives under
# dankserver/frank-primary/ except dated trash, and trash plans
# dankserver/frank-archon/ except dated trash, and trash plans
# nothing (the retire rule above the service blocks).
return services
@@ -400,32 +400,32 @@ def services_for_path(path: str, target: str) -> set[str]:
# change to it redeploying alertmanager is the only path that
# reaches it on the host.
_add(services, "alertmanager")
if name in {"frank-secondary.env", "frank-secondary.service", "serve-frank-secondary.sh"}:
# llama.cpp frank-secondary (2026-09-16) keeps its unit, env and serve
# files here; each one selects frank-secondary only. The ik_llama.cpp
if name in {"frank-zealot.env", "frank-zealot.service", "serve-frank-zealot.sh"}:
# llama.cpp frank-zealot (2026-09-16) keeps its unit, env and serve
# files here; each one selects frank-zealot only. The ik_llama.cpp
# trio that lived here for two days is in scripts/.AGENTTRASH/,
# which is archive material and plans nothing.
_add(services, "frank-secondary")
if name in {"frank-large.env", "frank-large.service", "serve-frank-large.sh"}:
# CIRU v4.4.1 frank-large (2026-09-19) keeps its unit, env and
_add(services, "frank-zealot")
if name in {"frank-dragoon.env", "frank-dragoon.service", "serve-frank-dragoon.sh"}:
# CIRU v4.4.1 frank-dragoon (2026-09-19) keeps its unit, env and
# serve files here the same way: a root unit on a host-built
# runtime. The download script fetches the payload, which is
# host-side state, so it selects nothing.
_add(services, "frank-large")
_add(services, "frank-dragoon")
if name == "qwen-sharp-v22.4.0.jinja":
# The Qwen-Sharp chat template frank-large loads via
# The Qwen-Sharp chat template frank-dragoon loads via
# --chat-template-file. It lived under dankserver/frank-medium/ until
# the 2026-09-26 cutover retired that directory; editing it changes
# what the live frank-large endpoint renders, so it plans
# frank-large.
_add(services, "frank-large")
if name in {"frank-primary.env", "frank-primary.service", "serve-frank-primary.sh"}:
# llama.cpp frank-primary (2026-09-26, replacing the Lucebox
# what the live frank-dragoon endpoint renders, so it plans
# frank-dragoon.
_add(services, "frank-dragoon")
if name in {"frank-archon.env", "frank-archon.service", "serve-frank-archon.sh"}:
# llama.cpp frank-archon (2026-09-26, replacing the Lucebox
# DS4.1 quadlet) keeps its unit, env and serve files here the
# same way: a root unit on the unsloth b11030-mix prebuilt
# serving Qwen3.8-Flash-Next Q8_0. The download script fetches
# the payload, which is host-side state, so it selects nothing.
_add(services, "frank-primary")
_add(services, "frank-archon")
# Deployment/test/benchmark scripts and other repo tooling do not
# restart a runtime service.
return services
+113 -113
View File
@@ -90,12 +90,12 @@ service_is_allowed() {
dankclaw:authentik-postgres|dankclaw:authentik-server|\
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
dankclaw:frank-large|dankclaw:frank-secondary|\
dankclaw:frank-dragoon|dankclaw:frank-zealot|\
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
dankserver:sonarr|dankserver:lidarr|dankserver:readarr|\
dankserver:frank-primary|dankserver:node-exporter|\
dankserver:frank-archon|dankserver:node-exporter|\
frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
frankbook-pro:readarr)
@@ -192,7 +192,7 @@ add_pending_services() {
continue
fi
[[ -n "$recorded" && "$recorded" != "$REVISION" ]] || continue
if [[ "$TARGET" == dankserver && "$service" == frank-large ]]; then continue; fi
if [[ "$TARGET" == dankserver && "$service" == frank-dragoon ]]; then continue; fi
if ! changed="$(python3 scripts/deploy-plan.py --target "$TARGET" \
--repo "$REMOTE_DIR" --base "$recorded" --head "$REVISION")"; then
echo "WARN: unable to compare state for $service; retaining it as pending" >&2
@@ -1092,21 +1092,21 @@ deploy_anubis() {
wait_container_healthy anubis 60 || return 1
}
# frank-large is the halogen-flash-server CONTAINER (0.16.2, digest-pinned
# through scripts/frank-large.env) serving the Quat3rnion ABLITERATED
# frank-dragoon is the halogen-flash-server CONTAINER (0.16.2, digest-pinned
# through scripts/frank-dragoon.env) serving the Quat3rnion ABLITERATED
# Swift1.5 finetune of peonist's Qwen3.8-Flash-Next v2 checkpoint —
# qwen38-flash-next-v2-swift15-abliterated.hgn (66,687,678,464 B, ~62 GiB
# resident) + the REQUIRED n-gram lookup table + the vision tower —
# MTP spec decode (HALOGEN_SPEC_ADAPT=0), 2 slots x 200192, image input —
# on the Strix Halo iGPU. It stays a ROOT unit because the serve script
# drives `podman run` directly (the frank-primary pattern): relink_unit
# points /etc/systemd/system/frank-large.service at the repo file, boot
# drives `podman run` directly (the frank-archon pattern): relink_unit
# points /etc/systemd/system/frank-dragoon.service at the repo file, boot
# persistence is that unit's [Install], and LimitMEMLOCK=infinity is
# fleet parity. 2026-10-06 cutover from the halogen 27B-dense shape —
# rollback = revert the swap commit's trio; the 27B payload stays on
# disk (~/models/halogen-qwen3.8-27b).
#
# The gates mirror serve-frank-large.sh's predicates EXACTLY (the
# The gates mirror serve-frank-dragoon.sh's predicates EXACTLY (the
# 2026-10-01 crash-loop lesson): image digest in dank's podman storage,
# each .hgn payload (checkpoint, n-gram table, vision tower) at the exact
# byte count, the tokenizer directory holding every named file as a
@@ -1114,23 +1114,23 @@ deploy_anubis() {
# artifact would leave a restart-looping unit and a dark endpoint. The
# port gate is the same as every cutover since 2026-09-16: nothing
# starts until 7770 is released.
LLM_LARGE_IMAGE="${LLM_LARGE_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a}"
LLM_LARGE_CKPT="${LLM_LARGE_CKPT:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-v2-swift15-abliterated.hgn}"
LLM_LARGE_CKPT_BYTES="${LLM_LARGE_CKPT_BYTES:-66687678464}"
LLM_LARGE_NGRAM="${LLM_LARGE_NGRAM:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-ngram.hgn}"
LLM_LARGE_NGRAM_BYTES="${LLM_LARGE_NGRAM_BYTES:-51200246144}"
LLM_LARGE_VISION="${LLM_LARGE_VISION:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-vision.hgn}"
LLM_LARGE_VISION_BYTES="${LLM_LARGE_VISION_BYTES:-897916416}"
LLM_LARGE_TOK_DIR="${LLM_LARGE_TOK_DIR:-/home/dank/models/halogen-flashnext-v2/tokenizer}"
LLM_LARGE_TOK_FILES="${LLM_LARGE_TOK_FILES:-tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json}"
LLM_DRAGOON_IMAGE="${LLM_DRAGOON_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a}"
LLM_DRAGOON_CKPT="${LLM_DRAGOON_CKPT:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-v2-swift15-abliterated.hgn}"
LLM_DRAGOON_CKPT_BYTES="${LLM_DRAGOON_CKPT_BYTES:-66687678464}"
LLM_DRAGOON_NGRAM="${LLM_DRAGOON_NGRAM:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-ngram.hgn}"
LLM_DRAGOON_NGRAM_BYTES="${LLM_DRAGOON_NGRAM_BYTES:-51200246144}"
LLM_DRAGOON_VISION="${LLM_DRAGOON_VISION:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-vision.hgn}"
LLM_DRAGOON_VISION_BYTES="${LLM_DRAGOON_VISION_BYTES:-897916416}"
LLM_DRAGOON_TOK_DIR="${LLM_DRAGOON_TOK_DIR:-/home/dank/models/halogen-flashnext-v2/tokenizer}"
LLM_DRAGOON_TOK_FILES="${LLM_DRAGOON_TOK_FILES:-tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json}"
retire_halogen_large_quadlet() {
retire_halogen_dragoon_quadlet() {
local trash="$HOME/.AGENTTRASH/20260919-orca-cutover/quadlet"
systemctl --user stop frank-large.service 2>/dev/null || true
if [[ -e "$quadlet_unit_dir/frank-large.container" ]]; then
systemctl --user stop frank-dragoon.service 2>/dev/null || true
if [[ -e "$quadlet_unit_dir/frank-dragoon.container" ]]; then
mkdir -p "$trash"
mv -- "$quadlet_unit_dir/frank-large.container" "$trash/frank-large.container"
echo "retired quadlet frank-large.container -> $trash"
mv -- "$quadlet_unit_dir/frank-dragoon.container" "$trash/frank-dragoon.container"
echo "retired quadlet frank-dragoon.container -> $trash"
fi
systemctl --user daemon-reload
}
@@ -1154,165 +1154,165 @@ wait_port_free() {
return 1
}
deploy_llm_large() {
deploy_llm_dragoon() {
# Refuse rather than restart-loop; every gate below has a twin in
# serve-frank-large.sh and they must agree (2026-10-01 lesson).
# serve-frank-dragoon.sh and they must agree (2026-10-01 lesson).
# Capture podman output before grepping: `podman images | grep -q`
# races under pipefail (SIGPIPE 141 becomes "no match", 2026-09-30).
if ! grep -qF "${LLM_LARGE_IMAGE#*@}" <<<"$(podman images --format '{{.RepoDigests}}' 2>/dev/null)"; then
echo "ERROR: halogen image absent: $LLM_LARGE_IMAGE" >&2
echo "frank-large: pull it: podman pull ${LLM_LARGE_IMAGE%@*}" >&2
if ! grep -qF "${LLM_DRAGOON_IMAGE#*@}" <<<"$(podman images --format '{{.RepoDigests}}' 2>/dev/null)"; then
echo "ERROR: halogen image absent: $LLM_DRAGOON_IMAGE" >&2
echo "frank-dragoon: pull it: podman pull ${LLM_DRAGOON_IMAGE%@*}" >&2
return 1
fi
if [[ ! -f "$LLM_LARGE_CKPT" ]]; then
echo "ERROR: halogen checkpoint absent at $LLM_LARGE_CKPT" >&2
echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
if [[ ! -f "$LLM_DRAGOON_CKPT" ]]; then
echo "ERROR: halogen checkpoint absent at $LLM_DRAGOON_CKPT" >&2
echo "frank-dragoon: fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
return 1
fi
if [[ "$(stat -c %s "$LLM_LARGE_CKPT")" != "$LLM_LARGE_CKPT_BYTES" ]]; then
echo "ERROR: halogen checkpoint is not $LLM_LARGE_CKPT_BYTES bytes (partial download?)" >&2
echo "frank-large: re-check: bash scripts/download-frank-large-flashnext.sh --verify-only" >&2
if [[ "$(stat -c %s "$LLM_DRAGOON_CKPT")" != "$LLM_DRAGOON_CKPT_BYTES" ]]; then
echo "ERROR: halogen checkpoint is not $LLM_DRAGOON_CKPT_BYTES bytes (partial download?)" >&2
echo "frank-dragoon: re-check: bash scripts/download-frank-dragoon-flashnext.sh --verify-only" >&2
return 1
fi
for gate in "$LLM_LARGE_NGRAM:$LLM_LARGE_NGRAM_BYTES:n-gram table" \
"$LLM_LARGE_VISION:$LLM_LARGE_VISION_BYTES:vision tower"; do
for gate in "$LLM_DRAGOON_NGRAM:$LLM_DRAGOON_NGRAM_BYTES:n-gram table" \
"$LLM_DRAGOON_VISION:$LLM_DRAGOON_VISION_BYTES:vision tower"; do
IFS=: read -r gpath gbytes gname <<<"$gate"
if [[ ! -f "$gpath" ]]; then
echo "ERROR: halogen $gname absent at $gpath" >&2
echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
echo "frank-dragoon: fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
return 1
fi
if [[ "$(stat -c %s "$gpath")" != "$gbytes" ]]; then
echo "ERROR: halogen $gname is not $gbytes bytes (partial download?)" >&2
echo "frank-large: re-check: bash scripts/download-frank-large-flashnext.sh --verify-only" >&2
echo "frank-dragoon: re-check: bash scripts/download-frank-dragoon-flashnext.sh --verify-only" >&2
return 1
fi
done
if [[ -L "$LLM_LARGE_TOK_DIR" || ! -d "$LLM_LARGE_TOK_DIR" ]]; then
echo "ERROR: tokenizer dir missing or a symlink at $LLM_LARGE_TOK_DIR" >&2
echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
if [[ -L "$LLM_DRAGOON_TOK_DIR" || ! -d "$LLM_DRAGOON_TOK_DIR" ]]; then
echo "ERROR: tokenizer dir missing or a symlink at $LLM_DRAGOON_TOK_DIR" >&2
echo "frank-dragoon: fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
return 1
fi
for tf in $LLM_LARGE_TOK_FILES; do
if [[ ! -f "$LLM_LARGE_TOK_DIR/$tf" || -L "$LLM_LARGE_TOK_DIR/$tf" ]]; then
echo "ERROR: tokenizer file missing or a symlink: $LLM_LARGE_TOK_DIR/$tf" >&2
echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
for tf in $LLM_DRAGOON_TOK_FILES; do
if [[ ! -f "$LLM_DRAGOON_TOK_DIR/$tf" || -L "$LLM_DRAGOON_TOK_DIR/$tf" ]]; then
echo "ERROR: tokenizer file missing or a symlink: $LLM_DRAGOON_TOK_DIR/$tf" >&2
echo "frank-dragoon: fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
return 1
fi
done
retire_halogen_large_quadlet
retire_halogen_dragoon_quadlet
# The predecessor on a re-deploy is THIS unit: stop it (ExecStop stops
# the container) before waiting on the port. The stray sweep is keyed
# on the FULL container name, which appears in the argv of a manual
# `podman run --name frank-large-halogen` / `podman wait` but NOT in
# `podman run --name frank-dragoon-halogen` / `podman wait` but NOT in
# this deployer's own command line (`bash deploy-remote.sh
# frank-large`) — so it cannot shoot itself in the foot (the fleet
# pkill law), and it can never match frank-secondary's llama-server.
run_privileged systemctl stop frank-large.service 2>/dev/null || true
pkill -f 'frank-large-halogen' 2>/dev/null || true
podman rm -f frank-large frank-large-halogen >/dev/null 2>&1 || true
# frank-dragoon`) — so it cannot shoot itself in the foot (the fleet
# pkill law), and it can never match frank-zealot's llama-server.
run_privileged systemctl stop frank-dragoon.service 2>/dev/null || true
pkill -f 'frank-dragoon-halogen' 2>/dev/null || true
podman rm -f frank-dragoon frank-dragoon-halogen >/dev/null 2>&1 || true
# Nothing may start until the predecessor has released the port
# (2026-09-16: "Deactivated successfully" was not "port free").
wait_port_free 7770 || return 1
relink_unit frank-large
relink_unit frank-dragoon
run_privileged systemctl daemon-reload
run_privileged systemctl enable frank-large.service
run_privileged systemctl restart frank-large.service
wait_for_health frank-large http://127.0.0.1:7770/health 1200
run_privileged systemctl enable frank-dragoon.service
run_privileged systemctl restart frank-dragoon.service
wait_for_health frank-dragoon http://127.0.0.1:7770/health 1200
}
# deploy_llm_secondary — frank-secondary is llama.cpp (master build, CUDA sm_86) on
# deploy_llm_zealot — frank-zealot is llama.cpp (master build, CUDA sm_86) on
# the RTX 3090 as a ROOT unit: the runtime is a native binary tree under
# ~/GitPackages/llama-master/build-cuda, so there is no container image and
# no generated unit to avoid. relink_unit points
# /etc/systemd/system/frank-secondary.service at the repo file and boot
# /etc/systemd/system/frank-zealot.service at the repo file and boot
# persistence is that unit's [Install] section.
#
# It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is
# not shared: frank-large's engine runs on the gfx1151 iGPU.
LLM_SECONDARY_MODEL_FILE="${LLM_SECONDARY_MODEL_FILE:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf}"
LLM_SECONDARY_RUNTIME="${LLM_SECONDARY_RUNTIME:-/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
# It is a SEPARATE endpoint on port 7771; frank-dragoon keeps 7770. The 3090 is
# not shared: frank-dragoon's engine runs on the gfx1151 iGPU.
LLM_ZEALOT_MODEL_FILE="${LLM_ZEALOT_MODEL_FILE:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf}"
LLM_ZEALOT_RUNTIME="${LLM_ZEALOT_RUNTIME:-/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
# The projector IS the vision capability here: the model is natively
# multimodal and the Q6_K file carries no vision tensors — the publisher ships
# its own BF16 projector in the same repo, so a deploy that starts without
# this file silently ships a text-only endpoint. Checked unconditionally, like
# frank-large's.
LLM_SECONDARY_MMPROJ="${LLM_SECONDARY_MMPROJ:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf}"
# frank-dragoon's.
LLM_ZEALOT_MMPROJ="${LLM_ZEALOT_MMPROJ:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf}"
deploy_llm_secondary() {
deploy_llm_zealot() {
# Refuse rather than restart-loop: a unit whose runtime or payload is
# missing never becomes healthy, and a green deploy would leave the
# endpoint dark with nothing in the log to say why.
if [[ ! -x "$LLM_SECONDARY_RUNTIME" ]]; then
echo "ERROR: llama.cpp runtime absent at $LLM_SECONDARY_RUNTIME" >&2
echo "frank-secondary: build it (cmake -B build-cuda -DGGML_CUDA=ON" >&2
echo "frank-secondary: -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13" >&2
echo "frank-secondary: && cmake --build build-cuda --target llama-server) first." >&2
if [[ ! -x "$LLM_ZEALOT_RUNTIME" ]]; then
echo "ERROR: llama.cpp runtime absent at $LLM_ZEALOT_RUNTIME" >&2
echo "frank-zealot: build it (cmake -B build-cuda -DGGML_CUDA=ON" >&2
echo "frank-zealot: -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13" >&2
echo "frank-zealot: && cmake --build build-cuda --target llama-server) first." >&2
return 1
fi
for f in "$LLM_SECONDARY_MODEL_FILE" "$LLM_SECONDARY_MMPROJ"; do
for f in "$LLM_ZEALOT_MODEL_FILE" "$LLM_ZEALOT_MMPROJ"; do
if [[ ! -f "$f" ]]; then
echo "ERROR: frank-secondary payload file absent at $f" >&2
echo "frank-secondary: run scripts/download-frank-secondary.sh first;" >&2
echo "frank-secondary: refusing to start a unit that cannot load." >&2
echo "ERROR: frank-zealot payload file absent at $f" >&2
echo "frank-zealot: run scripts/download-frank-zealot.sh first;" >&2
echo "frank-zealot: refusing to start a unit that cannot load." >&2
return 1
fi
done
relink_unit frank-secondary
relink_unit frank-zealot
run_privileged systemctl daemon-reload
run_privileged systemctl enable frank-secondary.service
run_privileged systemctl restart frank-secondary.service
wait_for_health frank-secondary http://127.0.0.1:7771/health 900 || return 1
# False-green hardening (frank-large's lesson): the port answering
run_privileged systemctl enable frank-zealot.service
run_privileged systemctl restart frank-zealot.service
wait_for_health frank-zealot http://127.0.0.1:7771/health 900 || return 1
# False-green hardening (frank-dragoon's lesson): the port answering
# proves nothing about WHO answers — the unit must be active with a
# live MainPID, or a stray SGLang holdover is masquerading as deployed.
local main_pid
main_pid="$(run_privileged systemctl show -p MainPID --value frank-secondary.service)"
main_pid="$(run_privileged systemctl show -p MainPID --value frank-zealot.service)"
if [[ -z "$main_pid" || "$main_pid" == "0" ]]; then
echo "FAIL: :7771 answered but frank-secondary.service has no MainPID — a stray process holds the port" >&2
echo "FAIL: :7771 answered but frank-zealot.service has no MainPID — a stray process holds the port" >&2
return 1
fi
echo "frank-secondary: health answered by unit MainPID $main_pid"
echo "frank-zealot: health answered by unit MainPID $main_pid"
}
# deploy_llm_primary — frank-primary is the radiance engine (StillDeadcode,
# deploy_llm_archon — frank-archon is the radiance engine (StillDeadcode,
# digest-pinned stilldeadcode/radiance image) serving the OFFICIAL
# Qwen/Qwen3.8-Flash-Next-FP8 at native block-FP8 precision from the
# rad-convert container qwen4exp-fp8-native.rad, on BOTH R9700s (TP=2) as a
# ROOT unit whose serve script drives `podman run` directly (the engine is
# the image entrypoint; the image lives in dank's rootless podman storage).
# 2026-10-05 cutover from FreeToken (pre-cutover tag
# frank-primary-freetoken-pr587-20261005); before that, the tcclaviger vLLM
# fork (tag frank-primary-vllm-shapeG-20261003). Same port and alias; the
# frank-archon-freetoken-pr587-20261005); before that, the tcclaviger vLLM
# fork (tag frank-archon-vllm-shapeG-20261003). Same port and alias; the
# harness models.yml id is unchanged.
#
# Gates: the image must exist under its digest ref in dank's rootless
# storage (FRANK_PRIMARY_IMAGE), the .rad container file must exist at the
# env's size floor (FRANK_PRIMARY_MODEL_FILE >= FRANK_PRIMARY_RAD_MIN_MIB —
# storage (FRANK_ARCHON_IMAGE), the .rad container file must exist at the
# env's size floor (FRANK_ARCHON_MODEL_FILE >= FRANK_ARCHON_RAD_MIN_MIB —
# a truncated convert is a guaranteed long-timeout boot), and the unit env
# must be complete. Per-file sha256 verification is the fetch script's job
# at download time and rad-convert/rad-info's job at build time, never
# re-hashed here.
LLM_PRIMARY_CONTAINER=frank-primary
LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
LLM_ARCHON_CONTAINER=frank-archon
LLM_ARCHON_LEGACY_CONTAINERS="frank-archon-qwen frank-archon-r9v qwen-fp8 glm-probe"
deploy_llm_primary() {
deploy_llm_archon() {
# Refuse rather than restart-loop: a unit whose image or payload is
# missing never becomes healthy, and a green deploy would leave the
# endpoint dark with nothing in the log to say why.
local env_file="/home/dank/serverAppConfigs/scripts/frank-primary.env"
local env_file="/home/dank/serverAppConfigs/scripts/frank-archon.env"
local image model_dir model_file rad_min mib
if [[ ! -f "$env_file" ]]; then
echo "ERROR: frank-primary env file absent: $env_file" >&2
echo "ERROR: frank-archon env file absent: $env_file" >&2
return 1
fi
image="$(sed -n 's/^FRANK_PRIMARY_IMAGE=//p' "$env_file" | tail -1)"
model_dir="$(sed -n 's/^FRANK_PRIMARY_MODEL_DIR=//p' "$env_file" | tail -1)"
model_file="$(sed -n 's/^FRANK_PRIMARY_MODEL_FILE=//p' "$env_file" | tail -1)"
rad_min="$(sed -n 's/^FRANK_PRIMARY_RAD_MIN_MIB=//p' "$env_file" | tail -1)"
image="$(sed -n 's/^FRANK_ARCHON_IMAGE=//p' "$env_file" | tail -1)"
model_dir="$(sed -n 's/^FRANK_ARCHON_MODEL_DIR=//p' "$env_file" | tail -1)"
model_file="$(sed -n 's/^FRANK_ARCHON_MODEL_FILE=//p' "$env_file" | tail -1)"
rad_min="$(sed -n 's/^FRANK_ARCHON_RAD_MIN_MIB=//p' "$env_file" | tail -1)"
if [[ -z "$image" || -z "$model_dir" || -z "$model_file" || -z "$rad_min" ]]; then
echo "ERROR: frank-primary.env missing FRANK_PRIMARY_IMAGE/MODEL_DIR/MODEL_FILE/RAD_MIN_MIB" >&2
echo "ERROR: frank-archon.env missing FRANK_ARCHON_IMAGE/MODEL_DIR/MODEL_FILE/RAD_MIN_MIB" >&2
return 1
fi
# The image gate is the digest ref itself: a name@sha256 ref exists in
@@ -1320,18 +1320,18 @@ deploy_llm_primary() {
# pipefail/grep race (the 2026-09-30 lesson).
if ! podman image exists "$image"; then
echo "ERROR: runtime image absent from storage: $image" >&2
echo "frank-primary: podman pull docker://stilldeadcode/radiance:latest" >&2
echo "frank-archon: podman pull docker://stilldeadcode/radiance:latest" >&2
return 1
fi
if [[ ! -f "$model_dir/$model_file" ]]; then
echo "ERROR: .rad container missing: $model_dir/$model_file" >&2
echo "frank-primary: run scripts/build-frank-primary-rad.sh" >&2
echo "frank-archon: run scripts/build-frank-archon-rad.sh" >&2
return 1
fi
mib=$(( $(stat -c %s "$model_dir/$model_file") / 1048576 ))
if [[ "$mib" -lt "$rad_min" ]]; then
echo "ERROR: .rad container too small: $model_dir/$model_file is ${mib} MiB, want >= $rad_min" >&2
echo "frank-primary: rebuild it (scripts/build-frank-primary-rad.sh --force)" >&2
echo "frank-archon: rebuild it (scripts/build-frank-archon-rad.sh --force)" >&2
return 1
fi
@@ -1343,19 +1343,19 @@ deploy_llm_primary() {
# The legacy identities (the retired R9V container, the ad hoc
# experiment container qwen-fp8, the retired glm-probe) hold the same
# GPUs and die here too.
run_privileged systemctl stop frank-primary.service 2>/dev/null || true
run_privileged systemctl stop frank-archon.service 2>/dev/null || true
local ctr
for ctr in $LLM_PRIMARY_LEGACY_CONTAINERS "$LLM_PRIMARY_CONTAINER"; do
for ctr in $LLM_ARCHON_LEGACY_CONTAINERS "$LLM_ARCHON_CONTAINER"; do
podman rm -f "$ctr" >/dev/null 2>&1 || true
done
# Nothing starts until the port is released: the lucebox cutover
# (2026-09-26) and the frank-large cutover (2026-09-16) both failed
# (2026-09-26) and the frank-dragoon cutover (2026-09-16) both failed
# exactly here when the next engine bound 7770 a second too early.
wait_port_free 7770 || return 1
relink_unit frank-primary
relink_unit frank-archon
run_privileged systemctl daemon-reload
run_privileged systemctl enable frank-primary.service
run_privileged systemctl restart frank-primary.service
run_privileged systemctl enable frank-archon.service
run_privileged systemctl restart frank-archon.service
# Cold start: .rad load + PLE pin + expert host-pool upload (the
# publisher's compose healthcheck allows a 15-minute start_period).
# Parity with every predecessor on this port. The HTTP listener binds
@@ -1366,14 +1366,14 @@ deploy_llm_primary() {
local i ok=0
for ((i=1; i<=900; i++)); do
if curl -fsS http://127.0.0.1:7770/health 2>/dev/null >/dev/null; then
echo "frank-primary engine healthy after approximately $((i * 2))s"
echo "frank-archon engine healthy after approximately $((i * 2))s"
ok=1
break
fi
sleep 2
done
if [[ "$ok" != 1 ]]; then
echo "FAIL: frank-primary /health never answered 200 in 1800s" >&2
echo "FAIL: frank-archon /health never answered 200 in 1800s" >&2
return 1
fi
# False-green hardening (the 2026-09-26 lesson): a bare curl against the
@@ -1381,12 +1381,12 @@ deploy_llm_primary() {
# it while the unit is stopped. The deploy is only green if the unit
# itself is active with a live MainPID behind the health answer.
local main_pid
main_pid="$(run_privileged systemctl show -p MainPID --value frank-primary.service)"
main_pid="$(run_privileged systemctl show -p MainPID --value frank-archon.service)"
if [[ -z "$main_pid" || "$main_pid" == "0" ]]; then
echo "FAIL: :7770 answered but frank-primary.service has no MainPID — a stray process holds the port" >&2
echo "FAIL: :7770 answered but frank-archon.service has no MainPID — a stray process holds the port" >&2
return 1
fi
echo "frank-primary: health answered by unit MainPID $main_pid"
echo "frank-archon: health answered by unit MainPID $main_pid"
}
wait_for_health() {
@@ -1573,9 +1573,9 @@ deploy_service() {
anubis) deploy_anubis ;;
seerr) deploy_seerr ;;
marreta) deploy_marreta ;;
frank-large) deploy_llm_large ;;
frank-secondary) deploy_llm_secondary ;;
frank-primary) deploy_llm_primary ;;
frank-dragoon) deploy_llm_dragoon ;;
frank-zealot) deploy_llm_zealot ;;
frank-archon) deploy_llm_archon ;;
hermes|hermes-dashboard|hermes-webui) deploy_hermes ;;
romm|romm-mariadb|romm-redis|mariadb|redis) deploy_romm_stack "$service" ;;
jellyfin) deploy_jellyfin ;;
+3 -3
View File
@@ -35,7 +35,7 @@ Options:
Examples:
./scripts/deploy.sh dankclaw
./scripts/deploy.sh dankclaw --service searxng
./scripts/deploy.sh dankserver --service frank-primary
./scripts/deploy.sh dankserver --service frank-archon
./scripts/deploy.sh dankclaw --dry-run
EOF
}
@@ -141,12 +141,12 @@ valid_service() {
dankclaw:alertmanager|\
dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
dankclaw:frank-large|dankclaw:frank-secondary|\
dankclaw:frank-dragoon|dankclaw:frank-zealot|\
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|\
dankserver:frank-primary|\
dankserver:frank-archon|\
frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
frankbook-pro:readarr)
@@ -1,6 +1,6 @@
#!/usr/bin/env bash
# download-frank-primary-swift15.sh — payload fetch/verify for the planned
# frank-primary swap to the Swift 1.5 finetune: ukisai/Swift1.5-Qwen3.8-Flash-Next
# download-frank-archon-swift15.sh — payload fetch/verify for the planned
# frank-archon swap to the Swift 1.5 finetune: ukisai/Swift1.5-Qwen3.8-Flash-Next
# (BF16 source, 149 files, 360,037,355,038 bytes = 335.3 GiB: 131 safetensors
# shards + weight-map index + tokenizer + vision preprocessing configs + chat
# template + licenses). Key set verified identical to the official
@@ -8,8 +8,8 @@
# apply verbatim; the checkpoint ships BF16 with NO quantization_config.
# Repo is UNGATED — no token needed (never put one on the command line).
#
# bash scripts/download-frank-primary-swift15.sh --fetch resume/complete
# bash scripts/download-frank-primary-swift15.sh verify only
# bash scripts/download-frank-archon-swift15.sh --fetch resume/complete
# bash scripts/download-frank-archon-swift15.sh verify only
#
# Fetch uses the vendored scripts/hfget.sh (StillDeadcode's Apache-2.0
# listing/resume downloader, curl+jq only): per-file size checks against
@@ -1,13 +1,13 @@
#!/usr/bin/env bash
# download-frank-primary.sh — payload fetch/verify for frank-primary:
# download-frank-archon.sh — payload fetch/verify for frank-archon:
# the OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint (131 safetensors
# shards + weight-map index + tokenizer + vision preprocessing configs,
# ~172.8 GB). 2026-10-05: replaces the orcarouter uncensored payload of the
# FreeToken era (that script lived here; its repo was gated and needed
# HF_TOKEN — this one is UNGATED and needs no token at all).
#
# bash scripts/download-frank-primary.sh --fetch resume/complete
# bash scripts/download-frank-primary.sh verify only
# bash scripts/download-frank-archon.sh --fetch resume/complete
# bash scripts/download-frank-archon.sh verify only
#
# Fetch uses the vendored scripts/hfget.sh (StillDeadcode's Apache-2.0
# listing/resume downloader, curl+jq only): per-file size checks against
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
# download-frank-large-flashnext.sh — fetch frank-large's flash-server v2
# download-frank-dragoon-flashnext.sh — fetch frank-dragoon's flash-server v2
# payload: the Quat3rnion abliterated Swift1.5 checkpoint of peonist's v2
# .hgn, the REQUIRED lookup table and the optional vision tower from
# peonist-ai/halogen-qwen3.8-flash-next (both at the revisions the model
@@ -29,7 +29,7 @@
# fetched fresh rather than symlinked, per the no-symlink law.
set -euo pipefail
DEST="${FRANK_LARGE_MODEL_DIR:-$HOME/models/halogen-flashnext-v2}"
DEST="${FRANK_DRAGOON_MODEL_DIR:-$HOME/models/halogen-flashnext-v2}"
ABLIT="https://huggingface.co/Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated/resolve/main"
PEONIST="https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next/resolve"
NGRAM_REV="5cc17cea1a10b502b4a5db41d8d2b1a0f22dba26"
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
# download-frank-large-halogen.sh — fetch frank-large's halogen payload:
# download-frank-dragoon-halogen.sh — fetch frank-dragoon's halogen payload:
# the qwen3.8-27b .hgn checkpoint and the export tokenizer, from the
# PUBLIC peonist-ai/halogen-qwen3.8-27b repo (no token — and per the fleet
# secret law, none would ever touch argv).
@@ -10,7 +10,7 @@
# existing payload without downloading.
set -euo pipefail
DEST="${FRANK_LARGE_MODEL_DIR:-$HOME/models/halogen-qwen3.8-27b}"
DEST="${FRANK_DRAGOON_MODEL_DIR:-$HOME/models/halogen-qwen3.8-27b}"
BASE="https://huggingface.co/peonist-ai/halogen-qwen3.8-27b/resolve/main"
FILES=(qwen3.8-27b-p1w4d-d2.hgn
@@ -1,6 +1,6 @@
#!/usr/bin/env bash
# Download kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF for
# frank-large (Dankclaw iGPU). Public repo — no auth header, and never a token
# frank-dragoon (Dankclaw iGPU). Public repo — no auth header, and never a token
# on the command line (it is world-readable in /proc/<pid>/cmdline).
#
# The payload is the repo's Q6 AGENT rung:
@@ -10,7 +10,7 @@
# names for MTP serving ("Take Q6 AGENT (114) if you want protected
# heads"). Like every ROCmFPX file it loads on nothing but the fork.
#
# The other two files frank-large needs are NOT re-downloaded:
# The other two files frank-dragoon needs are NOT re-downloaded:
# * mtp-Qwen3.8-27B-Q4_0.gguf — the uncensored repo redistributes the
# ALIGNED repo's draft head byte for byte (identical Hub LFS oid
# 051a1764…), so it is hardlinked out of the aligned payload dir when that
@@ -25,7 +25,7 @@
# byte count and the Hub LFS sha256. Idempotent: complete files are skipped.
#
# Usage:
# scripts/download-frank-large-kingjones-uncensored.sh [DEST_DIR] [--verify-only]
# scripts/download-frank-dragoon-kingjones-uncensored.sh [DEST_DIR] [--verify-only]
set -euo pipefail
REPO="kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF"
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
# Download kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF for frank-large
# Download kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF for frank-dragoon
# (Dankclaw iGPU). Public repo — no auth header, and never a token on the
# command line (it is world-readable in /proc/<pid>/cmdline).
#
@@ -24,7 +24,7 @@
# come from the Hub LFS oids (api/models/<repo>/tree/main).
#
# Usage:
# scripts/download-frank-large-kingjones.sh [DEST_DIR] [--verify-only]
# scripts/download-frank-dragoon-kingjones.sh [DEST_DIR] [--verify-only]
set -euo pipefail
REPO="kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF"
@@ -1,9 +1,9 @@
#!/usr/bin/env bash
# Download and assemble jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca (the frank-large
# Download and assemble jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca (the frank-dragoon
# model) on Dankclaw.
#
# bash scripts/download-frank-large-orca.sh
# bash scripts/download-frank-large-orca.sh --verify-only
# bash scripts/download-frank-dragoon-orca.sh
# bash scripts/download-frank-dragoon-orca.sh --verify-only
#
# The package is 136 GB of Hub-sized transport parts (HF caps new uploads at
# 50 GB, so the 79 GB main GGUF, the 52 GB PLE payload and the 4 GB MTP head
@@ -105,4 +105,4 @@ fi
echo "verifying assembled files against SHA256SUMS"
(cd "$MODEL_DIR" && sha256sum -c SHA256SUMS)
(cd "$MODEL_DIR/vision" && sha256sum -c SHA256SUMS)
echo "frank-large payload complete: $MODEL_DIR"
echo "frank-dragoon payload complete: $MODEL_DIR"
@@ -1,9 +1,9 @@
#!/usr/bin/env bash
# Download the Q4_K_XL frank-large payload on Dankclaw:
# Download the Q4_K_XL frank-dragoon payload on Dankclaw:
# unsloth/Qwen3.8-27B-GGUF -> Qwen3.8-27B-UD-Q4_K_XL.gguf
#
# bash scripts/download-frank-large-q4.sh
# bash scripts/download-frank-large-q4.sh --verify-only
# bash scripts/download-frank-dragoon-q4.sh
# bash scripts/download-frank-dragoon-q4.sh --verify-only
#
# Integrity: the file is checked against its declared byte count and sha256
# (the LFS oid from the Hub tree API, 2026-10-01). NO AUTH HEADER IS USED:
@@ -21,12 +21,12 @@
# x 2 (K,V) x 2 B), so 4 sessions x 163840 ctx = 640,000 tokens = 39.06 GiB.
#
# The speculative drafter is UNCHANGED and still fetched by its own script
# (scripts/download-frank-large.sh, z-lab's Hub repo). That name must not
# appear here at all — the frank-large seam greps THIS file for it to prove
# (scripts/download-frank-dragoon.sh, z-lab's Hub repo). That name must not
# appear here at all — the frank-dragoon seam greps THIS file for it to prove
# the drafter was not folded into the payload fetch.
#
# NOTE ON MTP: this payload has no baked-in MTP head (the Hub repo ships one
# separately as MTP/mtp-Qwen3.8-27B-Q4_0.gguf). frank-large does not need it —
# separately as MTP/mtp-Qwen3.8-27B-Q4_0.gguf). frank-dragoon does not need it —
# spec decode here is the block-diffusion drafter, not MTP.
set -euo pipefail
@@ -1,10 +1,10 @@
#!/usr/bin/env bash
# Download the Swift-1.5 frank-large candidate on Dankclaw:
# Download the Swift-1.5 frank-dragoon candidate on Dankclaw:
# ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF
# Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf
#
# bash scripts/download-frank-large-swift.sh
# bash scripts/download-frank-large-swift.sh --verify-only
# bash scripts/download-frank-dragoon-swift.sh
# bash scripts/download-frank-dragoon-swift.sh --verify-only
#
# Integrity: the file is checked against its declared byte count and sha256
# (the LFS oid from the Hub, 2026-09-28). NO AUTH HEADER IS USED: the repo is
@@ -15,9 +15,9 @@
# Layout after a successful run (MODEL_DIR):
# Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf 31457990272
#
# This is the CANDIDATE payload, not the deployed one. frank-large still
# serves scripts/download-frank-large.sh's output; do not repoint
# frank-large.env at MODEL_DIR until the gufo load test passes.
# This is the CANDIDATE payload, not the deployed one. frank-dragoon still
# serves scripts/download-frank-dragoon.sh's output; do not repoint
# frank-dragoon.env at MODEL_DIR until the gufo load test passes.
set -euo pipefail
MODEL_DIR="/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL"
@@ -1,9 +1,9 @@
#!/usr/bin/env bash
# Download unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (the
# frank-large model) on Dankclaw.
# frank-dragoon model) on Dankclaw.
#
# bash scripts/download-frank-large-unsloth.sh
# bash scripts/download-frank-large-unsloth.sh --verify-only
# bash scripts/download-frank-dragoon-unsloth.sh
# bash scripts/download-frank-dragoon-unsloth.sh --verify-only
#
# The UD-IQ4_XS GGUF ships as three llama.cpp split parts (HF caps uploads at
# 50 GB). The parts ARE the canonical payload: the serve script points -m at
@@ -112,4 +112,4 @@ if [[ "$mode" == "download" ]]; then
[[ -e "$MODEL_DIR/vision/mmproj-F16.gguf" ]] || mv "$MODEL_DIR/mmproj-F16.gguf" "$MODEL_DIR/vision/"
fi
echo "frank-large payload complete: $MODEL_DIR"
echo "frank-dragoon payload complete: $MODEL_DIR"
@@ -1,10 +1,10 @@
#!/usr/bin/env bash
# Download the frank-large payload on Dankclaw:
# Download the frank-dragoon payload on Dankclaw:
# - unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (the 27B dense main GGUF)
# - z-lab/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M (the DFlash2 drafter)
#
# bash scripts/download-frank-large.sh
# bash scripts/download-frank-large.sh --verify-only
# bash scripts/download-frank-dragoon.sh
# bash scripts/download-frank-dragoon.sh --verify-only
#
# Integrity: every file is checked against its declared byte count and sha256
# (the values below are the LFS oids from the Hub, 2026-09-21). An HF token
@@ -86,4 +86,4 @@ if [[ "$mode" == "download" ]]; then
done
fi
echo "frank-large payload complete: $MODEL_DIR"
echo "frank-dragoon payload complete: $MODEL_DIR"
@@ -1,5 +1,5 @@
#!/usr/bin/env bash
# Download the frank-secondary payload for llama.cpp on the RTX 3090:
# Download the frank-zealot payload for llama.cpp on the RTX 3090:
# HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at **Q6_K** plus the
# PUBLISHER's OWN vision projector (the BF16 mmproj — the model is natively
# multimodal, so this file is the endpoint's eyes). Public repo — no auth
@@ -21,7 +21,7 @@
# replaced), Swift15-UncIQ3XXS/, MiMo-Q6K/, MiMo-Q5KM/, Ornith-Q8/.
#
# Usage:
# scripts/download-frank-secondary.sh [DEST_DIR] [--verify-only]
# scripts/download-frank-zealot.sh [DEST_DIR] [--verify-only]
set -euo pipefail
DEST=""
@@ -1,5 +1,5 @@
#!/usr/bin/env python3
"""Measure per-block / per-expert expert-routing activity on live frank-primary.
"""Measure per-block / per-expert expert-routing activity on live frank-archon.
Method: AMD IBS op sampling (data addresses, `perf record -e ibs_op// --data`)
on the llama-server process while concurrent generations of mixed content run.
+286
View File
@@ -0,0 +1,286 @@
# frank-archon.env — radiance (StillDeadcode, Apache-2.0) serving
# ukisai/Swift1.5-Qwen3.8-Flash-Next converted to GPTQ w4 (MODEL_FILE below,
# built by scripts/convert-swift15-w4dn8.sh) on BOTH R9700s (TP=2),
# dankserver :7770.
#
# SHAPE (2026-10-13): THREE decode slots x 200,000 tokens, fp8 (quantized) KV,
# with the per-card budget spent on resident experts: 25,600 MiB of slab +
# 5,120 MiB of pool = 30,720 MiB, inside the 30,969 MiB ceiling the engine's own
# budget table allows (card 32,624 MiB, settle floor 209 MiB). The pool is sized
# to the contract and nothing more (3 x 200,000 needs 5,063 MiB at fp8; 5,120
# predicts 608,027 tokens, 1.3 % slack), because every MiB taken from the slab
# is prefill and TTFT (2026-10-11 A/B: slab -14.8 % cost prefill -29 %, TTFT
# +25 % at the full window) while KV beyond the contract buys nothing.
# Consequence, deliberate: a FOURTH full-height conversation does not fit this
# pair, and the engine refuses it — it never truncates. See the KV block for the
# arithmetic and the measured dtype law. The fp8-native payload this file was
# written for is the rollback target and keeps its own trio under the
# fp8-native runbook.
#
# 2026-10-05 cutover from FreeToken (PR #587 ROCm, one card, experts in
# pinned host RAM; git tag frank-archon-freetoken-pr587-20261005 is the
# pre-cutover state — its unit/env/serve trio and Dockerfile are all in the
# tag and rebuildable). Before that: tcclaviger vLLM fork (tag
# frank-archon-vllm-shapeG-20261003).
#
# WHY RADIANCE: the engine is C++/HIP with libr4d kernels built for
# gfx1201, drives `podman run` directly from the digest-pinned
# stilldeadcode/radiance image (287 MiB; host needs only the amdgpu driver),
# and its qwen4exp plugin reads the modelopt FP8 encoding NATIVELY
# (`X.weight_scale_inv` is the scale plane — checkpoint.h). The PLE n-gram
# table (51B params, 47.7 GiB) is served from pinned host RAM via io_uring
# reads — Frank's call: "Native FP8. Experts from ram. Pin the PLE to ram."
# The 4-bit-codebook publisher shape (qwen3.8-next-flash-fp8-iq4r-moe.rad,
# --host-pool-mib 12288) remains the documented fallback if the native
# container proves unbootable.
#
# PAYLOAD: /home/dank/models/rad/qwen4exp-fp8-native.rad — a rad-convert
# NO-RECIPE pure copy of the official 131-shard checkpoint (every weight
# keeps its block-FP8 encoding; ~161 GiB). The convert runs against a
# staging dir whose config.json drops `quantization_config`: the descriptor
# string is only the plugin SELECTION key (spec §2.4 — exact match, no
# fallback), and no shipped plugin claims (qwen4exp, "fp8"); the shipped
# qwen4exp plugin claims (qwen4exp, "") which is what recipe-built
# containers carry. Per-weight encodings are data, not the descriptor —
# removing the key changes nothing about what the weights hold. Built by
# scripts/build-frank-archon-rad.sh (staging + convert + rad-info).
#
# SERVING FACTS (docs/GUIDE.md, docs/DOCKER.md, compose/flashnext.yaml):
# - The HTTP listener binds AFTER the full load (main.cpp: engine.start()
# then engine.serve()) — during load the port refuses connections, so a
# bare curl /health gate is honest. /health == /ping: 200 JSON while
# serving, 503 only while stopping (core/server/admin.cpp:385).
# - Reasoning: --reasoning-format defaults to auto, emitting thinking in
# `reasoning_content` (config.cpp). Tool calls: the qwen3.8 dialect, from
# the chat format. Both need no parser flags.
# - Sampling: the OpenAI endpoint accepts temperature, top_p, top_k, min_p,
# typical_p, presence_penalty, frequency_penalty, repetition_penalty,
# repeat_last_n, chat_template_kwargs, reasoning_effort,
# enable_thinking, preserve_thinking (core/server/oai.cpp:194-205) —
# models.yml's extraBody set is fully inside it. Unknown params are
# refused (400), which is why nothing unverified rides the wire.
# - MTP: the official checkpoint carries the mtp.* head (3,101 tensors,
# mtp_num_hidden_layers=1) and the plugin probes draft depth 3 — the
# publisher's throughput peak for this model. Stated explicitly with
# --num-speculative-tokens 3.
# - The fused TP all-reduce is declared over max-num-seqs x
# (1 + --num-speculative-tokens) rows; 2 x 3 = 6 rows, under the 25-row
# exactness bound of the wht6 wire.
# Engine pin: Codeberg tag v1.0.13 (commit 53282fa), Docker Hub tag 1.0.13,
# index digest below. REJECTED UPGRADE: 1.2.0 (sha256:665775ed1d90de6dbb
# 25558e07fc60d20328f1346f96f3c6e00845eb763f01fa, tag 1.2.0, commit
# e95d397) was benched 2026-10-06 on this exact shape (fp8-native rad,
# vram pair, spec 2, seqs 3, 43.6k greedy probe): decode 29.21 t/s and
# draft acceptance 0.7123 vs the 1.0.13 control's 30.08 and 0.7549.
# Acceptance on the greedy harness is deterministic (the control
# reproduced the recorded 0.755 exactly), so the drop is a real numerics
# change — the 1.2.0 MoE weight-rotation change (92b8aaf) is the named
# suspect; decode loss tracks the acceptance ratio (0.943 vs 0.945).
# The vram-pair translation itself is exact (1.2.0 boot ledger took
# vram_weights 25.46 GiB / vram_kv 3.46 GiB per card byte-identical to
# the 1.0.13 ledger at EVC 0.82). Re-test 1.2.0+ when the rotation change
# is untangled; 1.2.0 also REMOVED --expert-vs-cache-ratio in favour of
# the --vram-weights-mib / --vram-kv-mib pair (core/config.cpp) — this
# file already uses the pair, which BOTH pins accept (rollback symmetry).
FRANK_ARCHON_IMAGE=stilldeadcode/radiance@sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
# The .rad container FILE (mounted under /models) and the state dir for the
# prefix-cache disk tier (mounted rw at /data). The model dir holds ONLY
# .rad files; the raw HF download and its staging dir live beside it and
# are never mounted.
FRANK_ARCHON_MODEL_DIR=/home/dank/models/rad
FRANK_ARCHON_MODEL_FILE=qwen4exp-swift15-w4dn8-i8-hc8m.rad
FRANK_ARCHON_STATE_DIR=/home/dank/radiance-state
# The state dir's tune/ subdirectory holds rad-tune's kernel-selection
# cache (the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune
# at boot; $RADIANCE_HOME defaults to /opt/radiance, in the image, so the
# cache must ride a host mount — serve script mounts it, see the runbook's
# "Tuning levers").
# Container identity is the SERVICE name (Frank's call, 2026-10-05): one
# name across systemctl and podman ps. The runtime changes; the name does
# not. The legacy list retires every predecessor identity that could hold
# the GPUs or the port.
FRANK_ARCHON_CONTAINER=frank-archon
FRANK_ARCHON_LEGACY_CONTAINERS="frank-archon-qwen frank-archon-r9v qwen-fp8 glm-probe"
# Served model id — Protoss name (wire rename 2026-10-09). The container,
# unit, scripts and this file keep the frank-archon name above; only the
# alias on the wire moved. Radiance validates the requested model name, so
# every client (harness, bench scripts, probes) MUST send this value.
FRANK_ARCHON_SERVED_NAME=frank-archon
FRANK_ARCHON_PORT=7770
# Frank's chat-template call: froggeric's fixed Qwen3.8 template (the
# repo's root chat_template.jinja, vendored here; the checkpoint's own
# template is baked into the .rad by rad-convert and this overrides it).
# Radiance derives the reasoning/tool-call reply format from it.
FRANK_ARCHON_TEMPLATE=/home/dank/serverAppConfigs/scripts/qwen38-froggeric-v22.5.jinja
# --- engine shape (the boot ledger arbitrates every number below) ---------
# Tensor parallel: BOTH R9700s. The board swap (2026-09-20, ProArt B850
# x8/x8 on root ports) made collectives legal on this host; radiance's TP
# wire is libr4d's own, not RCCL.
FRANK_ARCHON_TP=2
# Lossy WHT 6-bit wire for large all-reduces (publisher's production choice:
# +5.6-6.3% prefill, ~-1.3% 8-stream decode at TP2). min-kb stays the
# engine default.
FRANK_ARCHON_TP_WIRE=wht6
# Decode slots: THREE, so the served window is a CAPACITY CONTRACT — three
# concurrent conversations of --max-model-len fit the KV pool (see the KV
# block below). The "seqs 2 cannot boot" law is NOT radiance's: it was the
# tcclaviger vLLM fork's torch.compile shape specialization on
# query_start_loc (ConstraintViolationError, measured 2026-10-02, retired
# with that runtime). Radiance booted seqs 2 at the fp8-native first boot,
# and the 2026-10-06 decode sweep measured seqs 2 and 3 statistically tied
# (30.9-33.1 t/s, inside the +-3% noise floor) — slot count buys concurrency,
# not speed, so a slot costs KV, never decode.
# Cost of each extra slot: 110 MiB of per-slot recurrent state inside the KV
# pool, and the fused TP all-reduce is declared over seqs x (1 + spec) rows —
# 3 x 3 = 9 here, under the 25-row bound at which the wht6 wire stays EXACT
# (single-sequence decode and any step of <= 25 rows is bit-exact; only busier
# steps are lossy).
# The pool is sized to this contract and nothing more (see the KV block):
# 3 x 200,000 at fp8 = 5,063 MiB, pool 5,120, and the freed 1,792 MiB went into
# the resident expert slab. The trade is explicit and it is Frank's call:
# a FOURTH full-height conversation no longer fits (5,120 MiB of pool backs
# ~608k tokens; 4 x 200,000 needs 6,601), so a fourth 200k request is refused
# as too large to fit — it is NOT squeezed, paged, or truncated. Third-party
# traffic that wants four full-height sessions is a --vram-kv-mib change, not
# a code change.
FRANK_ARCHON_MAX_SEQS=3
# Served window. Native is 262,144; 200,000 is the publisher's production
# value (compose) and what models.yml's contextWindow must quote — a longer
# prompt is refused by the engine, never truncated. The engine treats
# `kv group tokens >= max-model-len` as a HARD limit ("ONE request longer
# than N tokens is accepted by the API and then refused as too large to ever
# fit — this is a hard limit, not a backing ratio"), so the pool is sized
# for seqs x maxlen, not for one sequence.
FRANK_ARCHON_MAX_MODEL_LEN=200000
FRANK_ARCHON_PLACEMENT=expert_tiered
# Pinned host pool for the expert rows that do not fit VRAM. vram_weights and
# this pool serve the per-rank resident set TOGETHER: at the fp8-KV pair
# (26071/3543) the ledger showed the VRAM slab filled to 259 KiB of slack,
# i.e. the pair was exactly sized. Cutting vram-weights by 3.8 GiB to pay for
# the bf16 KV pool SPILLS ~3.8 GiB/rank here, so the pool moves with it —
# under-sizing it is a boot failure (36864 refused the boot in the fp8 era).
# The boot ledger's coverage line remains the referee, not this arithmetic.
FRANK_ARCHON_HOST_POOL_MIB=44928
# VRAM left unclaimed per card (publisher default 96 MiB).
FRANK_ARCHON_GPU_HEADROOM_MIB=96
# VRAM split of claimed memory per card, MiB (0 = derive from the card).
# THE CEILING, read off this engine's own VRAM-budget table (boot 2026-10-13,
# pair 24000/5120): card 32,624 MiB; budgets claimed 29,120; `3.42 GiB is
# neither budgeted nor accounted — the driver's context, the activation arena
# and --gpu-headroom-mib come out of it; anything past those is throughput on
# the floor`; and `card 0 settled at 2.01 GiB free against 209.19 MiB (the
# 96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the static region)`.
# So the reserve above the ENGINE's floor is 2,058 - 209 = 1,849 MiB and the
# ceiling is 29,120 + 1,849 = 30,969 MiB. The 2026-10-11 bound of 29,120
# ("31,856 - 1,710 outside both budgets - keep 1 GiB free") was MY reserve, not
# the engine's; the pair below spends the difference and lands 458 MiB above
# the engine's floor. `weights + kv <= 30,969` is the gate.
# THE PRICE, and it is paid twice: on this engine idle KV VRAM is ALSO the
# expert cache's spillover (`mover: 2275 slots can be lent out of the KV
# cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <= 256 tokens`),
# so a bigger KV pool steals weight residency from the slab AND from the
# lendable cache. Priors on this card/model: stealing 2.14 GiB of weights for
# a bigger pool (MAX_MODEL_LEN 262144) cost 2.5% decode; the EVC ladder's
# other direction (0.82 -> 0.84, +2.5 GiB weights) cost 8.9%. The 2026-10-11
# exact-KV A/B measured the slab's own price (22208 vs the fp8-era 26071):
# prefill -29 %, TTFT at the full window +25 %, decode inside the noise band on
# 2 of 3 passes, draft acceptance UP 0.661 -> 0.740.
# 2026-10-13 (late): THREE slots, so the pool is cut to the contract (5,120)
# and every spare MiB goes to the slab: 22208 -> 24000 (the third slot's
# surplus KV), then 24000 -> 25600 (the card's settling slack). The direction
# the A/B data points: prefill is slab-limited, KV beyond the contract is not.
# The margin is thin by design (608,027 predicted tokens against the 600,000
# contract), so the boot ledger's `backs N sequences all at 200000 tokens at
# once` line is the referee for THIS pair and the concurrent probe is its
# regression test.
# VERDICT on the 25,600 arm (measured 2026-10-13, scripts/probe-frank-archon-kv.py):
# the threshold claim is FALSIFIED — the slab is a SLOPE at this end. A-cold
# 43.5k single-stream: 22,208 -> 3,397, 24,000 -> 3,290, 25,600 -> 3,584 t/s;
# the 3-way 197k arm's aggregate prefill 3,051 -> 3,589 t/s (wall
# 193.7 -> 164.5 s). So +1,600 MiB of slab = +8.9 % single-stream and
# +17.6 % concurrent, both outside the +-3 % noise floor. The 22,208/24,000
# pair landed inside noise of EACH OTHER, which is what made it read as a
# plateau; two points are not a curve.
# The cliff that remains is between 25,600 (3,584) and 26,071 (4,379, the
# 2026-10-11 fp8 control at seqs 3 / pool 3,543) — +22 % for +471 MiB. At THREE
# slots that is only reachable by breaking the ceiling (26,112 + 5,120 = 31,232
# vs 30,969, ~214 MiB above the engine's floor); at TWO slots it is free:
# pool 3,072 + slab 27,897 = exactly the ceiling, slab above the cliff.
FRANK_ARCHON_VRAM_WEIGHTS_MIB=25600
FRANK_ARCHON_VRAM_KV_MIB=5120
# KV cache QUANTIZED (fp8). This engine's flag takes ONLY bf16|fp8 — there is
# ONE kv_dtype for the whole cache, so "exact K, quantized V" (llama.cpp's
# --cache-type-k/--cache-type-v pair, which is what frank-zealot runs) is
# UNSERVABLE here: the rung below bf16 is fp8 on both halves.
# Measured per-rank cost, read out of the boot ledger's group table (fp8 boot
# 2026-10-08, bf16 boot 2026-10-11):
# kv_attn 13 layers, 4-token blocks, page 26.00 KiB (fp8) -> 52.00 KiB
# kv_qsa_bkey 13 layers, 4-token blocks, page 3.25 KiB at BOTH dtypes —
# the QSA block key is an index, not attention state, and does
# NOT follow --kv-cache-dtype
# => 7.3125 KiB/token fp8, 13.8125 KiB/token bf16: bf16 costs 1.89x fp8,
# not 2x, precisely because the bkey plane is dtype-fixed.
# The dtype-INDEPENDENT terms also live in the pool: kv_gdn_state 54 MiB x 2
# copies per sequence (+ conv/tail/ple ~= 110 MiB a sequence) and the prefix
# checkpoint region 8 x 55.99 = 448 MiB. So:
# pool_mib >= tokens x 7.3125 KiB + 110 x seqs + 448
# 3 x 200,000 at fp8 needs 4,285 + 330 + 448 = 5,063 MiB; the pool is 5,120,
# and it predicts (5120 - 330 - 448) x 1024 / 7.3125 = 608,027 tokens: the
# three-slot contract with 1.3 % slack, every MiB above 5,063 having been
# handed to the expert slab instead. The ledger line is the referee ("backs N
# sequences all at 200000 tokens at once") — a pair this tight is read off the
# boot, never assumed.
# At bf16 the same three slots cost 8,093 + 778 = 8,871 MiB, which would cut
# the slab to 20,249 MiB (-9 %); at FOUR slots bf16 costs 10,791 + 888 =
# 11,679 MiB and the slab falls to 17,441 (-33 %). fp8 keeps 22,208 at both
# slot counts. The 2026-10-11 bf16 shape (6,912 / seqs 2) backed 462,752
# tokens = 2 x 200,000; the historical fp8 pool of 3,543 gave 386,904 (1.93
# sessions), and the four-slot fp8 boot of 2026-10-13 19:21 backed 843,276.
# Secondary saving: the prefix-cache entry shrinks with the attention plane
# (one entry 110.50 KiB at bf16 -> 58.50 KiB at fp8, measured on the two boot
# ledgers' `kv tier transfer` lines), so the host tier below parks twice as
# many conversations as it did at bf16 — no re-size needed to park three slots.
FRANK_ARCHON_KV_DTYPE=fp8
# Spec decode: the checkpoint's own MTP head at the publisher's depth-3
# operating point (probe() returns 3 when mtp_num_hidden_layers > 0; the
# flag states it so the ledger records the intent).
FRANK_ARCHON_SPEC_TOKENS=2
FRANK_ARCHON_MAX_BATCHED_TOKENS=2048
# Prefix cache tiers (publisher pattern). The host tier is sized in ENTRIES,
# and one entry tracks the cache dtype (58.50 KiB at fp8, 117 KiB at bf16, per
# the ledger's `kv tier transfer` line): at fp8, 8192 MiB parks ~5.8 sessions
# of 200k (at bf16 the same tier parked ~2.9), so four live slots plus a full
# parked set fits with the tier as it stands — at RAM the host has (104 GiB
# available with the service loaded).
FRANK_ARCHON_PREFIX_HOST_MIB=8192
FRANK_ARCHON_PREFIX_DISK_MIB=65536
# Frank's pin law: the PLE n-gram table (47.7 GiB) serves from pinned host
# RAM the cards read. `ram` REFUSES the start if the memlock limit is too
# small — the unit's LimitMEMLOCK=infinity is the other half of this law.
FRANK_ARCHON_NGRAM_PLACEMENT=ram
# TTM preflight: the pinned ledger (2 x 44 GiB pools + 47.7 GiB PLE + 8 GiB
# prefix host = ~147 GiB) rides TTM's shared pages, capped by ttm's
# pages_limit (the current kernel exposes NO pages_max; the cap function
# takes the min over the knobs that exist). Below the plan the load dies
# mid-pin with an allocator error instead of this named refusal.
# THE CAP IS A RUNTIME KNOB — the kernel default is half of RAM, so every
# raise dies with a reboot. Measured the hard way: the 2026-10-05 manual raise
# to 52,428,800 pages (200 GiB) was gone after the 2026-10-12 23:17 POWER
# FAILURE, the cap fell back to 125,988 MiB, and the preflight refused the
# start 6,919 times in ~20 h while Restart=always re-ran it. The gate in
# scripts/ttm-preflight.sh now RAISES the knob itself before refusing (the
# root unit is privileged; the knob is a 0644 root module parameter), so the
# recovery is a boot, not a visit to the machine.
FRANK_ARCHON_TTM_MIN_MIB=150000
FRANK_ARCHON_TTM_TARGET_MIB=204800
# Escape hatch for boot-ladder experiments. Unset -> no extra argv. Keep
# multi-flag values as one string; the serve script expands it unquoted on
# purpose.
FRANK_ARCHON_EXTRA_ARGS=
# The .rad's minimum size in MiB (deployer gate; a truncated convert is a
# guaranteed long-timeout boot instead of a fast failure). The converted
# container measured 177,109 MiB (1,857,127,997,440 B = 173.0 GiB; the
# plan-only estimate 172.76 GiB covers the data planes but not the metadata
# tail) - floor sits just under the payload.
FRANK_ARCHON_RAD_MIN_MIB=110000
@@ -1,8 +1,8 @@
# frank-primary.service — radiance (StillDeadcode, Apache-2.0) endpoint
# frank-archon.service — radiance (StillDeadcode, Apache-2.0) endpoint
# (OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 at NATIVE block-FP8 precision, the
# rad-convert container qwen4exp-fp8-native.rad) on BOTH R9700s (TP=2,
# wht6 wire), dankserver :7770. 2026-10-05 cutover from FreeToken (git tag
# frank-primary-freetoken-pr587-20261005 is the pre-cutover state; the
# frank-archon-freetoken-pr587-20261005 is the pre-cutover state; the
# retired unit/env/serve trio and Dockerfile are in that tag).
#
# Same port, same served alias, same container identity as the unit name;
@@ -19,7 +19,7 @@
# unit grants LimitMEMLOCK=infinity to the dank process via PID1; the
# engine still runs out of dank's rootless podman storage.
[Unit]
Description=frank-primary — radiance Qwen3.8-Flash-Next-FP8 (native) on 2x R9700 TP=2
Description=frank-archon — radiance Qwen3.8-Flash-Next-FP8 (native) on 2x R9700 TP=2
After=network-online.target
Wants=network-online.target
# Never let the start-rate limiter give up after a crash loop (a failed LLM
@@ -33,14 +33,21 @@ Group=dank
# RLIMIT_MEMLOCK smaller than the pinned set fails the load mid-start.
# PID1 grants infinity here — parity with the fleet's other LLM units.
LimitMEMLOCK=infinity
Environment="FRANK_PRIMARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-primary.env"
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-primary.sh
Environment="FRANK_ARCHON_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-archon.env"
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-archon.sh
# The `+` prefix runs this as root even though the service is User=dank — the
# ttm pages cap is a root-owned knob and the pinned-RAM ledger (~147 GiB) is
# unusable below it. After a reboot the knob is back at its default (half of
# RAM), so the raise must be part of the start, not a note in a runbook: the
# 2026-10-12 power failure reset it and the endpoint refused to boot 6,919
# times. The serve script re-verifies the cap after this hook.
ExecStartPre=+/home/dank/serverAppConfigs/scripts/ttm-preflight.sh --raise
# systemd stop: wait alone cannot stop the container child; stop it
# explicitly. No graceful drain contract is claimed (the OpenAI front end
# holds no queue to drain — in-flight decodes are lost, which a stop
# during a bench window accepts; radiance takes SIGTERM on the engine
# process and podman reaps the container).
ExecStop=/usr/bin/podman stop -t 120 frank-primary
ExecStop=/usr/bin/podman stop -t 120 frank-archon
# Boot cost: .rad load (weights + PLE upload) + expert host-pool pin. The
# publisher's compose healthcheck allows a 15-minute start_period; the
# vLLM-era budget (1800 s) carried forward.
@@ -1,11 +1,11 @@
# frank-large.env — tunables for the frank-large endpoint on Dankclaw's
# frank-dragoon.env — tunables for the frank-dragoon endpoint on Dankclaw's
# Strix Halo iGPU (gfx1151), host port 7770.
#
# RUNTIME (2026-10-06, Frank): halogen-flash-server 0.16.2 — the
# closed-source C++/HIP engine (image ghcr.io/peonist-ai/halogen-flash-server,
# digest-pinned below) serving the Quat3rnion ABLITERATED Swift1.5 finetune
# of peonist's Qwen3.8-Flash-Next v2 checkpoint, as a podman container under
# a root unit, driven by scripts/serve-frank-large.sh. This CUTS OVER the
# a root unit, driven by scripts/serve-frank-dragoon.sh. This CUTS OVER the
# 2026-10-05 halogen 27B-dense shape (censored base, DFlash2, 31.7 t/s):
# rollback = revert the swap commit's trio; the 27B payload stays on disk at
# /home/dank/models/halogen-qwen3.8-27b/.
@@ -36,7 +36,7 @@
# target). Worst-case request: prompt + budget <= 217088 − 4096 + 3 %
# est-error ≈ 219k <= 262144 — the 45k gap between harness window and
# engine ceiling absorbs the estimate error with ~7x margin (robust to a
# 10 % under-count). Compaction rides on frank-secondary: input ≈
# 10 % under-count). Compaction rides on frank-zealot: input ≈
# threshold − keepRecent(20k) ≈ 170k measured, inside its 200,704 slot.
# HALOGEN_KV_POOL_POSITIONS=524288 = KV_SLOTS(2) x CTX — the explicit
# form of the engine's default, so the seam test can pin the product.
@@ -52,7 +52,7 @@
# weights ~62 GiB resident + KV 14.0 + arena (HALOGEN_MAX_TOK=8192 =>
# ~8 GiB per FLAGS.md arena formula 3.8+8.4*(MT/16384)) + vision ~2
# = ~86 GiB engine-side; HALOGEN_HOST_RESERVE_GIB=20 keeps the host
# honest; baseline co-tenancy (frank-secondary, ComfyUI, Caddy/observ)
# honest; baseline co-tenancy (frank-zealot, ComfyUI, Caddy/observ)
# ~10-15. ~15 GiB margin. The boot ledger is the referee — read it
# after deploy and re-size if the engine prints a clamp.
# HALOGEN_SPEC_ADAPT=0 is MANDATORY (card): the default adaptive policy
@@ -91,47 +91,47 @@
# Image: digest-pinned (fetch law: tag pulls silently re-point; the digest
# is what was qualified). 0.16.2 = the version the model card names as its
# quickstart pin; RepoDigest captured at pull on 2026-10-06.
FRANK_LARGE_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
FRANK_DRAGOON_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
# Container identity: kept from the 27B era (unit ExecStop names it).
FRANK_LARGE_CONTAINER=frank-large-halogen
FRANK_DRAGOON_CONTAINER=frank-dragoon-halogen
# Old quadlet-era container identity; the serve script clears stale names.
FRANK_LARGE_LEGACY_CONTAINERS=frank-large
FRANK_DRAGOON_LEGACY_CONTAINERS=frank-dragoon
# host:container — the frontend listens on 8731 inside (HALOGEN_API_PORT
# default); the engine port (8730) is never published.
FRANK_LARGE_PUBLISH=7770:8731
FRANK_DRAGOON_PUBLISH=7770:8731
FRANK_LARGE_MODEL_DIR=/home/dank/models/halogen-flashnext-v2
FRANK_DRAGOON_MODEL_DIR=/home/dank/models/halogen-flashnext-v2
# The checkpoint: Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated
# qwen38-flash-next-v2-swift15-abliterated.hgn — compacted r2 of the card's
# append-only original (byte-identical live tensors, padding dropped:
# 66,687,678,464 B vs 89,775,868,032 B). LFS sha256 verified at fetch.
FRANK_LARGE_CHECKPOINT=qwen38-flash-next-v2-swift15-abliterated.hgn
FRANK_LARGE_CHECKPOINT_BYTES=66687678464
FRANK_LARGE_CHECKPOINT_SHA256=9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037
FRANK_DRAGOON_CHECKPOINT=qwen38-flash-next-v2-swift15-abliterated.hgn
FRANK_DRAGOON_CHECKPOINT_BYTES=66687678464
FRANK_DRAGOON_CHECKPOINT_SHA256=9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037
# The n-gram lookup table: REQUIRED (the engine reads it beside the
# checkpoint; the card's quickstart names it explicitly). From
# peonist-ai/halogen-qwen3.8-flash-next @ 5cc17cea.
FRANK_LARGE_NGRAM_TABLE=qwen38-flash-next-ngram.hgn
FRANK_LARGE_NGRAM_BYTES=51200246144
FRANK_LARGE_NGRAM_SHA256=3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe
FRANK_DRAGOON_NGRAM_TABLE=qwen38-flash-next-ngram.hgn
FRANK_DRAGOON_NGRAM_BYTES=51200246144
FRANK_DRAGOON_NGRAM_SHA256=3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe
# The vision tower: OPTIONAL on the engine, ENABLED here (image input).
# From peonist-ai/halogen-qwen3.8-flash-next @ e053f488.
FRANK_LARGE_VISION_TOWER=qwen38-flash-next-vision.hgn
FRANK_LARGE_VISION_BYTES=897916416
FRANK_LARGE_VISION_SHA256=d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10
# The tokenizer is a FLAT DIRECTORY, $FRANK_LARGE_MODEL_DIR/tokenizer,
FRANK_DRAGOON_VISION_TOWER=qwen38-flash-next-vision.hgn
FRANK_DRAGOON_VISION_BYTES=897916416
FRANK_DRAGOON_VISION_SHA256=d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10
# The tokenizer is a FLAT DIRECTORY, $FRANK_DRAGOON_MODEL_DIR/tokenizer,
# mounted whole at /tokenizer. The two repos ship the same one; fetched
# fresh from peonist @ e053f488 (tokenizer.json is LFS, byte-identical to
# the 27B-era copy; the rest are plain git blobs — size-checked only). The
# serve gate and the deployer refuse a missing file or ANY symlink: an HF
# cache snapshot symlinks each file into blobs/, and those links dangle
# inside the container mount.
FRANK_LARGE_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json"
FRANK_DRAGOON_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json"
# --- engine flags (docs/FLAGS.md of the flash-server repo; anything unset
# stays at the image default, which is what was qualified) ------------------
# Served model id — Protoss name (wire rename 2026-10-09); the service, unit,
# scripts and this file keep the frank-large name. OpenWebUI api_configs
# scripts and this file keep the frank-dragoon name. OpenWebUI api_configs
# model_ids for the :7770 backend MUST name this value.
HALOGEN_MODEL_ID=frank-dragoon
HALOGEN_CTX=262144
@@ -1,5 +1,5 @@
[Unit]
Description=halogen-flash-server container (Quat3rnion abliterated Swift1.5 of Qwen3.8-Flash-Next v2, MTP spec decode, 2 slots x 200192, vision ON), root systemd unit (dankclaw, frank-large, Strix Halo iGPU)
Description=halogen-flash-server container (Quat3rnion abliterated Swift1.5 of Qwen3.8-Flash-Next v2, MTP spec decode, 2 slots x 200192, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
After=network-online.target
Wants=network-online.target
@@ -11,11 +11,11 @@ Group=dank
# RLIMIT smaller than the pinned set cannot fail the load. Parity with the
# other LLM units on this fleet.
LimitMEMLOCK=infinity
Environment="FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env"
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh
Environment="FRANK_DRAGOON_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-dragoon.env"
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-dragoon.sh
# systemd stop alone cannot stop the container child; stop it explicitly
# (frank-primary pattern). The engine drains on SIGTERM.
ExecStop=/usr/bin/podman stop -t 60 frank-large-halogen
# (frank-archon pattern). The engine drains on SIGTERM.
ExecStop=/usr/bin/podman stop -t 60 frank-dragoon-halogen
TimeoutStartSec=1200
TimeoutStopSec=120
# always, not on-failure: an engine that exits 0 after an internal fault
@@ -23,7 +23,7 @@ TimeoutStopSec=120
Restart=always
RestartSec=10
# The engine pins ~62 GiB of v2 weights and keeps a multi-GiB working set
# on a box that also hosts frank-secondary, ComfyUI and the Caddy/observability
# on a box that also hosts frank-zealot, ComfyUI and the Caddy/observability
# stack; make it a late OOM candidate, not the first.
OOMScoreAdjust=-500
-168
View File
@@ -1,168 +0,0 @@
# frank-primary.env — radiance (StillDeadcode, Apache-2.0) serving the
# OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint at NATIVE precision
# (modelopt block-FP8 throughout, no requantisation) on BOTH R9700s (TP=2),
# dankserver :7770.
#
# 2026-10-05 cutover from FreeToken (PR #587 ROCm, one card, experts in
# pinned host RAM; git tag frank-primary-freetoken-pr587-20261005 is the
# pre-cutover state — its unit/env/serve trio and Dockerfile are all in the
# tag and rebuildable). Before that: tcclaviger vLLM fork (tag
# frank-primary-vllm-shapeG-20261003).
#
# WHY RADIANCE: the engine is C++/HIP with libr4d kernels built for
# gfx1201, drives `podman run` directly from the digest-pinned
# stilldeadcode/radiance image (287 MiB; host needs only the amdgpu driver),
# and its qwen4exp plugin reads the modelopt FP8 encoding NATIVELY
# (`X.weight_scale_inv` is the scale plane — checkpoint.h). The PLE n-gram
# table (51B params, 47.7 GiB) is served from pinned host RAM via io_uring
# reads — Frank's call: "Native FP8. Experts from ram. Pin the PLE to ram."
# The 4-bit-codebook publisher shape (qwen3.8-next-flash-fp8-iq4r-moe.rad,
# --host-pool-mib 12288) remains the documented fallback if the native
# container proves unbootable.
#
# PAYLOAD: /home/dank/models/rad/qwen4exp-fp8-native.rad — a rad-convert
# NO-RECIPE pure copy of the official 131-shard checkpoint (every weight
# keeps its block-FP8 encoding; ~161 GiB). The convert runs against a
# staging dir whose config.json drops `quantization_config`: the descriptor
# string is only the plugin SELECTION key (spec §2.4 — exact match, no
# fallback), and no shipped plugin claims (qwen4exp, "fp8"); the shipped
# qwen4exp plugin claims (qwen4exp, "") which is what recipe-built
# containers carry. Per-weight encodings are data, not the descriptor —
# removing the key changes nothing about what the weights hold. Built by
# scripts/build-frank-primary-rad.sh (staging + convert + rad-info).
#
# SERVING FACTS (docs/GUIDE.md, docs/DOCKER.md, compose/flashnext.yaml):
# - The HTTP listener binds AFTER the full load (main.cpp: engine.start()
# then engine.serve()) — during load the port refuses connections, so a
# bare curl /health gate is honest. /health == /ping: 200 JSON while
# serving, 503 only while stopping (core/server/admin.cpp:385).
# - Reasoning: --reasoning-format defaults to auto, emitting thinking in
# `reasoning_content` (config.cpp). Tool calls: the qwen3.8 dialect, from
# the chat format. Both need no parser flags.
# - Sampling: the OpenAI endpoint accepts temperature, top_p, top_k, min_p,
# typical_p, presence_penalty, frequency_penalty, repetition_penalty,
# repeat_last_n, chat_template_kwargs, reasoning_effort,
# enable_thinking, preserve_thinking (core/server/oai.cpp:194-205) —
# models.yml's extraBody set is fully inside it. Unknown params are
# refused (400), which is why nothing unverified rides the wire.
# - MTP: the official checkpoint carries the mtp.* head (3,101 tensors,
# mtp_num_hidden_layers=1) and the plugin probes draft depth 3 — the
# publisher's throughput peak for this model. Stated explicitly with
# --num-speculative-tokens 3.
# - The fused TP all-reduce is declared over max-num-seqs x
# (1 + --num-speculative-tokens) rows; 3 x 3 = 9 rows fits either wire.
# Engine pin: Codeberg tag v1.0.13 (commit 53282fa), Docker Hub tag 1.0.13,
# index digest below. REJECTED UPGRADE: 1.2.0 (sha256:665775ed1d90de6dbb
# 25558e07fc60d20328f1346f96f3c6e00845eb763f01fa, tag 1.2.0, commit
# e95d397) was benched 2026-10-06 on this exact shape (fp8-native rad,
# vram pair, spec 2, seqs 3, 43.6k greedy probe): decode 29.21 t/s and
# draft acceptance 0.7123 vs the 1.0.13 control's 30.08 and 0.7549.
# Acceptance on the greedy harness is deterministic (the control
# reproduced the recorded 0.755 exactly), so the drop is a real numerics
# change — the 1.2.0 MoE weight-rotation change (92b8aaf) is the named
# suspect; decode loss tracks the acceptance ratio (0.943 vs 0.945).
# The vram-pair translation itself is exact (1.2.0 boot ledger took
# vram_weights 25.46 GiB / vram_kv 3.46 GiB per card byte-identical to
# the 1.0.13 ledger at EVC 0.82). Re-test 1.2.0+ when the rotation change
# is untangled; 1.2.0 also REMOVED --expert-vs-cache-ratio in favour of
# the --vram-weights-mib / --vram-kv-mib pair (core/config.cpp) — this
# file already uses the pair, which BOTH pins accept (rollback symmetry).
FRANK_PRIMARY_IMAGE=stilldeadcode/radiance@sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
# The .rad container FILE (mounted under /models) and the state dir for the
# prefix-cache disk tier (mounted rw at /data). The model dir holds ONLY
# .rad files; the raw HF download and its staging dir live beside it and
# are never mounted.
FRANK_PRIMARY_MODEL_DIR=/home/dank/models/rad
FRANK_PRIMARY_MODEL_FILE=qwen4exp-swift15-w4dn8-i8-hc8m.rad
FRANK_PRIMARY_STATE_DIR=/home/dank/radiance-state
# The state dir's tune/ subdirectory holds rad-tune's kernel-selection
# cache (the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune
# at boot; $RADIANCE_HOME defaults to /opt/radiance, in the image, so the
# cache must ride a host mount — serve script mounts it, see the runbook's
# "Tuning levers").
# Container identity is the SERVICE name (Frank's call, 2026-10-05): one
# name across systemctl and podman ps. The runtime changes; the name does
# not. The legacy list retires every predecessor identity that could hold
# the GPUs or the port.
FRANK_PRIMARY_CONTAINER=frank-primary
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
# Served model id — Protoss name (wire rename 2026-10-09). The container,
# unit, scripts and this file keep the frank-primary name above; only the
# alias on the wire moved. Radiance validates the requested model name, so
# every client (harness, bench scripts, probes) MUST send this value.
FRANK_PRIMARY_SERVED_NAME=frank-archon
FRANK_PRIMARY_PORT=7770
# Frank's chat-template call: froggeric's fixed Qwen3.8 template (the
# repo's root chat_template.jinja, vendored here; the checkpoint's own
# template is baked into the .rad by rad-convert and this overrides it).
# Radiance derives the reasoning/tool-call reply format from it.
FRANK_PRIMARY_TEMPLATE=/home/dank/serverAppConfigs/scripts/qwen38-froggeric-v22.5.jinja
# --- engine shape (first-boot conservative; the boot ledger arbitrates) --
# Tensor parallel: BOTH R9700s. The board swap (2026-09-20, ProArt B850
# x8/x8 on root ports) made collectives legal on this host; radiance's TP
# wire is libr4d's own, not RCCL.
FRANK_PRIMARY_TP=2
# Lossy WHT 6-bit wire for large all-reduces (publisher's production choice:
# +5.6-6.3% prefill, ~-1.3% 8-stream decode at TP2). min-kb stays the
# engine default.
FRANK_PRIMARY_TP_WIRE=wht6
# Decode slots. 2 is Frank's first-boot value ("use parallelism = 2 to
# start"); raise only after a real-traffic look at slot occupancy and the
# KV pool's headroom line. The recurrent-state cost scales with it (54
# MiB/copy/card, 2 copies with speculation).
FRANK_PRIMARY_MAX_SEQS=3
# Served window. Native is 262,144; 200,000 is the publisher's production
# value (compose) and what models.yml's contextWindow must quote — a longer
# prompt is refused by the engine, never truncated.
FRANK_PRIMARY_MAX_MODEL_LEN=200000
FRANK_PRIMARY_PLACEMENT=expert_tiered
# Pinned host pool for the expert rows that do not fit VRAM. Native FP8
# experts are 112.8 GiB total (56.4 GiB per rank slice) vs the 4-bit
# publisher shape's ~29 GiB — this pool is the cost of native precision.
# The boot ledger prints the pool's coverage ("the pool backs N% of the
# worst case ... M full-length sessions"); resize from that line, not from
# arithmetic.
FRANK_PRIMARY_HOST_POOL_MIB=40960
# VRAM left unclaimed per card (publisher default 96 MiB).
FRANK_PRIMARY_GPU_HEADROOM_MIB=96
# VRAM split of claimed memory per card, MiB (0 = derive from the card).
# Engine 1.2.0 REMOVED --expert-vs-cache-ratio; these two knobs replace
# it. The values transcribe the live 1.0.13 boot ledger at ratio 0.82:
# vram_weights 25.46 GiB, vram_kv 3.46 GiB per card -> 26071 / 3543.
# Bracket arms scale the KV term linearly in (1-r) from the 0.82 anchor
# with weights fixed: ratio 0.90 -> KV 1968, ratio 0.95 -> KV 984. The
# boot ledger arbitrates; re-derive after any payload change.
FRANK_PRIMARY_VRAM_WEIGHTS_MIB=26071
FRANK_PRIMARY_VRAM_KV_MIB=3543
# KV cache in FP8 (7.32 KiB/token at this model's 12 full-attention layers,
# 2 KV heads, 256 head_dim, K+V).
FRANK_PRIMARY_KV_DTYPE=fp8
# Spec decode: the checkpoint's own MTP head at the publisher's depth-3
# operating point (probe() returns 3 when mtp_num_hidden_layers > 0; the
# flag states it so the ledger records the intent).
FRANK_PRIMARY_SPEC_TOKENS=2
FRANK_PRIMARY_MAX_BATCHED_TOKENS=2048
# Prefix cache tiers (publisher pattern; disk tier sized for 2 slots).
FRANK_PRIMARY_PREFIX_HOST_MIB=4096
FRANK_PRIMARY_PREFIX_DISK_MIB=65536
# Frank's pin law: the PLE n-gram table (47.7 GiB) serves from pinned host
# RAM the cards read. `ram` REFUSES the start if the memlock limit is too
# small — the unit's LimitMEMLOCK=infinity is the other half of this law.
FRANK_PRIMARY_NGRAM_PLACEMENT=ram
# TTM preflight: the pinned ledger (2 x 40 GiB pools + 47.7 GiB PLE + 4 GiB
# prefix host) is ~132 GiB. ttm's pages_limit/pages_max cap the GPU-mappable
# pinned pages; the current cap (32,252,917 pages = ~123 GiB) is BELOW the
# plan, so the serve script refuses to start until both are raised. This is
# the floor it enforces (in MiB), with ~8 GiB of margin.
FRANK_PRIMARY_TTM_MIN_MIB=140000
# Escape hatch for boot-ladder experiments. Unset -> no extra argv. Keep
# multi-flag values as one string; the serve script expands it unquoted on
# purpose.
FRANK_PRIMARY_EXTRA_ARGS=
# The .rad's minimum size in MiB (deployer gate; a truncated convert is a
# guaranteed long-timeout boot instead of a fast failure). The converted
# container measured 177,109 MiB (1,857,127,997,440 B = 173.0 GiB; the
# plan-only estimate 172.76 GiB covers the data planes but not the metadata
# tail) - floor sits just under the payload.
FRANK_PRIMARY_RAD_MIN_MIB=110000
@@ -1,4 +1,4 @@
# frank-secondary — llama.cpp (master, CUDA sm_86) serving
# frank-zealot — llama.cpp (master, CUDA sm_86) serving
# HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090,
# :7771.
#
@@ -38,7 +38,7 @@ MMPROJ_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensore
# Served model id — the harness/family name (Protoss rename completed to the
# wire 2026-10-09: harness ids + homepage cards 2026-10-08, /v1/models aliases
# this day). The SERVICE, unit, scripts and this env file keep the
# frank-secondary name; only the alias on the wire changed. Consumers that
# frank-zealot name; only the alias on the wire changed. Consumers that
# key on the served id: OpenWebUI (model row, access_grant, api_configs
# model_ids) and the harness models.yml.
ALIAS=frank-zealot
@@ -1,5 +1,5 @@
[Unit]
Description=llama.cpp Qwen3.5-9B-Uncensored-HauhauCS-Aggressive Q6_K GGUF (qwen35 hybrid, 2x200704 slots, f16 K+V, MTP off, mmproj vision) on the RTX 3090 (dankclaw, frank-secondary)
Description=llama.cpp Qwen3.5-9B-Uncensored-HauhauCS-Aggressive Q6_K GGUF (qwen35 hybrid, 2x200704 slots, f16 K+V, MTP off, mmproj vision) on the RTX 3090 (dankclaw, frank-zealot)
After=network-online.target
Wants=network-online.target
@@ -10,8 +10,8 @@ Group=dank
# CUDA pins its staging buffers, so the process needs the memlock budget even
# though the weights themselves are copied into VRAM.
LimitMEMLOCK=infinity
Environment="FRANK_SECONDARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-secondary.env"
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-secondary.sh
Environment="FRANK_ZEALOT_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-zealot.env"
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-zealot.sh
Restart=on-failure
RestartSec=10
+1 -1
View File
@@ -24,7 +24,7 @@ KLD_OUT=/home/dank/radiance-state/kld-gate.json
: > "$KLD_OUT"
cp -f /home/dank/serverAppConfigs/scripts/qwen38-froggeric-v22.5.jinja \
/home/dank/models/tpl.jinja 2>/dev/null || true
sudo systemctl stop frank-primary >/dev/null 2>&1 || true
sudo systemctl stop frank-archon >/dev/null 2>&1 || true
podman rm -f "$C" >/dev/null 2>&1 || true
podman run --rm --name "$C" \
+5 -5
View File
@@ -44,17 +44,17 @@ JSONL_EXTS = (".jsonl", ".json")
DOC_DIR = "serverAppConfigs-docs"
FLEET_CMDS = [
"systemctl status frank-primary --no-pager",
"journalctl -u frank-primary -n 100 --no-pager",
"systemctl status frank-archon --no-pager",
"journalctl -u frank-archon -n 100 --no-pager",
"podman ps --format '{{.Names}} {{.Status}}'",
"podman logs --tail 50 frank-primary",
"podman logs --tail 50 frank-archon",
"curl -s http://192.168.0.235:7770/health",
"curl -s http://192.168.0.235:7770/metrics | grep radiance_",
"df -h /home/dank | tail -1",
"free -g",
"sudo systemctl restart frank-primary",
"sudo systemctl restart frank-archon",
"cd /home/dank/serverAppConfigs && git pull --ff-only",
"./scripts/deploy.sh dankserver --service frank-primary",
"./scripts/deploy.sh dankserver --service frank-archon",
"./tests/run.sh 2>&1 | tail -5",
"jq . /home/dank/models/rad/index.json | head -40",
"python3 -c \"import json; print(json.load(open('config.json'))['num_hidden_layers'])\"",
@@ -2,7 +2,7 @@
#
# STATUS: WIP — first of three patches in the "compile the GLM trunk" changelist. The
# set does NOT yet boot the engine (see 03 for the next blocker), so it is NOT part of
# the deployed frank-primary shape. Nothing in scripts/ mounts these files.
# the deployed frank-archon shape. Nothing in scripts/ mounts these files.
#
# WHY THIS IS THE RIGHT LEVER (measured 2026-09-30, not inferred):
#
+1 -1
View File
@@ -1,6 +1,6 @@
# gpu_worker-kv-cap.patch — fork-local fix for the expert-offload KV cap.
#
# STATUS: NOT part of the deployed frank-primary shape. It exists because the
# STATUS: NOT part of the deployed frank-archon shape. It exists because the
# fork's plan under-models this model's KV, and it is the enabler for a future
# MTP attempt (the production GLM shape runs without MTP and boots fine on the
# plan's own reserve).
+201
View File
@@ -0,0 +1,201 @@
#!/usr/bin/env python3
"""probe-frank-archon-kv.py — verify the seqs x max-model-len KV contract on a
frank-archon (radiance) service. Run it ON dankserver: it talks to
127.0.0.1:7770 and builds its prompts from the repo checkout there.
No scratch container, no mask: the service under test is the service.
ssh dankserver 'python3 /home/dank/serverAppConfigs/scripts/probe-frank-archon-kv.py'
Four probes, in increasing cost:
A decode at depth — one ~51k-token chat completion, 512 tokens, greedy.
Reports the engine's own `timings` (prefill, decode, draft acceptance).
The bench law applies: >=128 generated tokens, greedy, real text.
B the capacity contract — as many CONCURRENT chats of ~197k prompt tokens as
the deployed `FRANK_ARCHON_MAX_SEQS` (read from the repo env file on this
host), each carrying a unique marker at its head. All must answer their own
marker. This is what the endpoint owes: N full-height conversations
resident at once, exact retrieval in every one of them.
C prefix-cache restore — probe A's cold prompt again, so the disk tier has
to give the blocks back (cache_n > 0, TTFT collapses).
D concurrency decode — `seqs` CONCURRENT 512-token generations at ~25 % of
the window. This is the cost a slot count imposes on DECODE, which no
prefill number shows: the fused-step schedule is unfair, so every stream is
printed plus the aggregate. Measured 2026-10-13 at 3 x 200,000 fp8:
single-stream 99.0 t/s (43.5k) and 93.3 (197k) — decode is nearly
window-free — against 3-stream 113.9/43.3/10.9 t/s = 18.4 t/s aggregate.
Concurrency, not the slab, is what decode pays for slots. The 2026-10-13
shape hunt missed this for a whole day because leg B generates 24 tokens.
PROBE LAW (learned the hard way, twice): radiance routes the thinking block to
`reasoning_content`, so a chat request whose budget is spent inside
`<think>` returns EMPTY `content` with `finish_reason: stop`. A probe that
reads `content` alone scores 0/2 on a healthy server. Every request here sets
chat_template_kwargs.enable_thinking=false and reads both fields.
A/B method for a shape change (no scratch container): write a variant env file
on the host, point a transient systemd drop-in at it
(`Environment=FRANK_ARCHON_ENV_FILE=...` — drop-ins are read AFTER the unit's
own Environment= lines, so they win), restart, probe, delete the drop-in,
daemon-reload, restart. `systemctl mask` is not available: the unit file in
/etc/systemd/system is a relink symlink into the checkout.
"""
import json
import glob
import time
import urllib.request
import concurrent.futures
BASE = "http://127.0.0.1:7770"
MODEL = "frank-archon"
REPO = "/home/dank/serverAppConfigs"
GLOBS = [f"{REPO}/scripts/*.sh", f"{REPO}/serverAppConfigs-docs/**/*.md"]
def post(payload, timeout=1800):
req = urllib.request.Request(
BASE + "/v1/chat/completions",
data=json.dumps(payload).encode(),
headers={"Content-Type": "application/json"},
)
t0 = time.time()
with urllib.request.urlopen(req, timeout=timeout) as r:
body = json.loads(r.read().decode())
hdr = dict(r.headers.items())
return body, time.time() - t0, hdr
def get(path, timeout=60):
with urllib.request.urlopen(BASE + path, timeout=timeout) as r:
return json.loads(r.read().decode())
def corpus(min_chars):
files = []
for g in GLOBS:
files.extend(sorted(glob.glob(g, recursive=True)))
out, total = [], 0
for f in files:
try:
t = open(f, encoding="utf-8", errors="replace").read()
except OSError:
continue
out.append(t)
total += len(t)
if total > min_chars:
break
return "".join(text for text in out)
def prompt(text, chars, salt, fact):
"""Real repo text, salted so the prefix cache cannot answer, marker planted
near the head, question at the tail."""
head = f"\n[SALT {salt}] confidential marker: {fact}\n"
body = text + text
p = (head + body * max(1, chars // len(body)))[:chars]
return p + "\n\nQuestion: what is the confidential marker in the " \
"SALT block near the top of this document? Answer with the marker " \
"alone, no other words.\n"
def chat(text, chars, salt, fact, max_tokens):
return {"model": MODEL,
"messages": [{"role": "user",
"content": prompt(text, chars, salt, fact)}],
"max_tokens": max_tokens, "temperature": 0.0,
"chat_template_kwargs": {"enable_thinking": False}}
def show(tag, j, wall, expect=None):
tim, u = j.get("timings") or {}, j.get("usage", {})
ch = (j.get("choices") or [{}])[0]
msg = ch.get("message") or {}
txt = (msg.get("content") or msg.get("reasoning_content") or "").strip()
acc = (tim["draft_n_accepted"] / tim["draft_n"]) if tim.get("draft_n") else None
line = (f"[{tag}] prompt={u.get('prompt_tokens')} gen={u.get('completion_tokens')} "
f"wall={wall:.1f}s prefill={tim.get('prompt_per_second', 0):.0f} t/s "
f"decode={tim.get('predicted_per_second', 0):.1f} t/s "
f"draft={tim.get('draft_n')}/{tim.get('draft_n_accepted')} "
f"accept={acc if acc is None else round(acc, 4)} "
f"cache_n={tim.get('cache_n')} finish={ch.get('finish_reason')}")
if expect:
hit = expect.split("-")[0] in txt.lower()
line += f" expected={expect} answered={txt[:80]!r} HIT={hit}"
print(line)
def deployed_shape():
"""The contract is the deployed env, not a constant: slot count and window
come from the checkout on this host, so the probe cannot drift from the
service it is testing (the 2026-10-13 shape is 4 x 200,000; before that
2 x 200,000, and the fp8 era 3 x 200,000)."""
shape = {}
with open(f"{REPO}/scripts/frank-archon.env") as fh:
for line in fh:
line = line.strip()
if line.startswith("FRANK_ARCHON_") and "=" in line:
key, _, val = line.partition("=")
shape[key] = val.strip().strip('"')
return (int(shape["FRANK_ARCHON_MAX_SEQS"]),
int(shape["FRANK_ARCHON_MAX_MODEL_LEN"]),
shape["FRANK_ARCHON_KV_DTYPE"])
def main():
seqs, maxlen, dtype = deployed_shape()
print(f"shape: {seqs} x {maxlen:,} tokens, kv {dtype}")
print("health:", json.dumps(get("/health")))
print("models:", [m.get("id") for m in get("/v1/models").get("data", [])])
deep = corpus(160_000)
wide = corpus(1_100_000)
# 3.28 chars/token is the measured ratio of this corpus through the
# froggeric template (646,000 chars -> 196,919 tokens at the 2026-10-11
# deploy); 98.5 % of the window leaves the response room under the ceiling.
chars = int(0.985 * maxlen * 3.28)
# Each marker's FIRST hyphen-separated word is unique, because show()
# scores the hit on that word — a shared prefix would score every slot HIT
# on any answer that says the word.
words = ["cobalt", "quartz", "basalt", "ember", "willow", "cobweb", "flint"]
facts = {f"B{i+1}": f"{words[i % len(words)]}-flag-{(i + 1) * 137}"
for i in range(seqs)}
bodies = {s: chat(wide, chars, s, f, 24) for s, f in facts.items()}
print(f"[B] {len(bodies['B1']['messages'][0]['content']):,} chars per "
f"request, x{seqs} concurrent")
# A + C first (cheap), B last (the expensive set).
for tag, salt in (("A-cold", "A1-9137"), ):
j, wall, _ = post(chat(deep, 140_000, salt, "zephyr-lantern-7", 512))
show(tag, j, wall, "zephyr-lantern-7")
t0 = time.time()
with concurrent.futures.ThreadPoolExecutor(max_workers=seqs) as ex:
futs = {s: ex.submit(post, b) for s, b in bodies.items()}
for s, fut in futs.items():
try:
j, wall, _ = fut.result()
show(f"B-{s}", j, wall, facts[s])
except Exception as e: # a preemption/timeout lands here
print(f"[B-{s}] FAILED after {time.time() - t0:.1f}s: {e!r}")
print(f"[B] total {time.time() - t0:.1f}s")
# D — concurrency decode: seqs streams, 512 tokens each, ~25 % depth.
dchars = int(0.25 * maxlen * 3.28)
dbodies = [chat(deep, dchars, f"D{i + 1}-9137", f"flint-marker-{i}", 512)
for i in range(seqs)]
print(f"[D] {seqs} concurrent streams x 512 tokens, "
f"{dchars:,} chars each")
t0 = time.time()
with concurrent.futures.ThreadPoolExecutor(max_workers=seqs) as ex:
futs = [ex.submit(post, b) for b in dbodies]
for i, fut in enumerate(futs, 1):
try:
j, wall, _ = fut.result()
show(f"D-s{i}", j, wall)
except Exception as e: # a preemption/timeout lands here
print(f"[D-s{i}] FAILED after {time.time() - t0:.1f}s: {e!r}")
tot = time.time() - t0
print(f"[D] {seqs} x 512 tokens in {tot:.1f}s = "
f"{seqs * 512 / tot:.1f} t/s aggregate decode")
j, wall, _ = post(chat(deep, 140_000, "A1-9137", "zephyr-lantern-7", 512))
show("C-prefix-hit", j, wall, "zephyr-lantern-7")
if __name__ == "__main__":
main()
@@ -1,6 +1,6 @@
# qwen4exp-swift15-w4dn8-i8-hc8m.recipe -- the Swift 1.5 finetune of Qwen3.8-Flash-Next
# (ukisai/Swift1.5-Qwen3.8-Flash-Next, BF16 source, 335.3 GiB, key set identical to the official
# base) as a radiance container for frank-primary: EVERY routed expert -- gate_up AND down --
# base) as a radiance container for frank-archon: EVERY routed expert -- gate_up AND down --
# through GPTQ into the w4nl64a8h table. The plugin's one-table-one-encoding law
# ("one table of experts takes one encoding, and only a whole expert kept plain bf16 is served
# beside it", qwen4exp_fp8 rad_arch_declare) forbids the original split tier (down block-fp8 +
+2 -2
View File
@@ -30,9 +30,9 @@ KLD_REF_DIR=/home/dank/corpora/kld-ref
CALIB_DIR=/home/dank/corpora/calib
KLD_CORPUS=/home/dank/corpora/out/kld.jsonl
restore() { sudo systemctl start frank-primary >/dev/null 2>&1 || true; }
restore() { sudo systemctl start frank-archon >/dev/null 2>&1 || true; }
trap restore EXIT
sudo systemctl stop frank-primary >/dev/null 2>&1 || true
sudo systemctl stop frank-archon >/dev/null 2>&1 || true
podman rm -f "$C" >/dev/null 2>&1 || true
ARGS=(--model /models/rad/qwen4exp-swift15-bf16.rad
+196
View File
@@ -0,0 +1,196 @@
#!/usr/bin/env bash
# serve-frank-archon.sh — frank-archon endpoint: radiance (StillDeadcode,
# Apache-2.0) serving the OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint at
# NATIVE block-FP8 precision from a rad-convert container
# (/home/dank/models/rad/qwen4exp-fp8-native.rad), on BOTH R9700s (TP=2,
# wht6 wire), dankserver :7770. 2026-10-05 cutover from FreeToken (PR #587,
# one card; git tag frank-archon-freetoken-pr587-20261005 is the
# pre-cutover state).
#
# The run line below IS the engine argv: the image's entrypoint is the
# engine and its tools sit beside it on PATH. Unlike every predecessor
# there is no per-boot VRAM planner to second-guess: the boot ledger in the
# log states every budget before anything is allocated (weights vs KV
# split, the expert host pool's coverage of the worst case, the PLE pin,
# the P2P answer per pair) — read it after every resize.
# Deployed shape (2026-10-13): THREE decode slots x 200,000 tokens, fp8
# (quantized) KV in a 5,120 MiB pool (contract 3 x 200,000 + 1.3 %) and a
# 25,600 MiB resident expert slab — 30,720 MiB of the 30,969 MiB the engine's
# own budget table makes claimable, spent on the slab because prefill is
# slab-limited. 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache host
# 8 GiB + disk 64 GiB, PLE pinned in RAM.
#
# The root unit (scripts/frank-archon.service, User=dank,
# LimitMEMLOCK=infinity) grants the pin budget: 2 x 44 GiB expert pools +
# the 47.7 GiB PLE table + 8 GiB prefix host tier — ~147 GiB of pinned
# pages, which is ALSO why this script refuses to start when ttm's
# pages_limit/pages_max cap is under FRANK_ARCHON_TTM_MIN_MIB (see below).
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${FRANK_ARCHON_ENV_FILE:-$SCRIPT_DIR/frank-archon.env}"
if [ ! -f "$ENV_FILE" ]; then
echo "ERROR: frank-archon env file absent: $ENV_FILE" >&2
exit 1
fi
set -a
# shellcheck disable=SC1090
source "$ENV_FILE"
set +a
: "${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
: "${FRANK_ARCHON_MODEL_DIR:?FRANK_ARCHON_MODEL_DIR must name the .rad dir}"
: "${FRANK_ARCHON_MODEL_FILE:?FRANK_ARCHON_MODEL_FILE must name the .rad file}"
: "${FRANK_ARCHON_STATE_DIR:?FRANK_ARCHON_STATE_DIR must name the /data state dir}"
: "${FRANK_ARCHON_CONTAINER:?FRANK_ARCHON_CONTAINER must name the container}"
: "${FRANK_ARCHON_SERVED_NAME:?FRANK_ARCHON_SERVED_NAME must name the alias}"
: "${FRANK_ARCHON_PORT:?FRANK_ARCHON_PORT must be the port}"
: "${FRANK_ARCHON_TEMPLATE:?FRANK_ARCHON_TEMPLATE must name the chat template}"
: "${FRANK_ARCHON_TP:?FRANK_ARCHON_TP must be set}"
: "${FRANK_ARCHON_TP_WIRE:?FRANK_ARCHON_TP_WIRE must be set}"
: "${FRANK_ARCHON_MAX_SEQS:?FRANK_ARCHON_MAX_SEQS must be set}"
: "${FRANK_ARCHON_MAX_MODEL_LEN:?FRANK_ARCHON_MAX_MODEL_LEN must be set}"
: "${FRANK_ARCHON_PLACEMENT:?FRANK_ARCHON_PLACEMENT must be set}"
: "${FRANK_ARCHON_HOST_POOL_MIB:?FRANK_ARCHON_HOST_POOL_MIB must be set}"
: "${FRANK_ARCHON_GPU_HEADROOM_MIB:?FRANK_ARCHON_GPU_HEADROOM_MIB must be set}"
: "${FRANK_ARCHON_VRAM_WEIGHTS_MIB:?FRANK_ARCHON_VRAM_WEIGHTS_MIB must be set}"
: "${FRANK_ARCHON_VRAM_KV_MIB:?FRANK_ARCHON_VRAM_KV_MIB must be set}"
: "${FRANK_ARCHON_KV_DTYPE:?FRANK_ARCHON_KV_DTYPE must be set}"
: "${FRANK_ARCHON_SPEC_TOKENS:?FRANK_ARCHON_SPEC_TOKENS must be set}"
: "${FRANK_ARCHON_MAX_BATCHED_TOKENS:?FRANK_ARCHON_MAX_BATCHED_TOKENS must be set}"
: "${FRANK_ARCHON_PREFIX_HOST_MIB:?FRANK_ARCHON_PREFIX_HOST_MIB must be set}"
: "${FRANK_ARCHON_PREFIX_DISK_MIB:?FRANK_ARCHON_PREFIX_DISK_MIB must be set}"
: "${FRANK_ARCHON_NGRAM_PLACEMENT:?FRANK_ARCHON_NGRAM_PLACEMENT must be set}"
: "${FRANK_ARCHON_TTM_MIN_MIB:?FRANK_ARCHON_TTM_MIN_MIB must be set}"
: "${FRANK_ARCHON_TTM_TARGET_MIB:?FRANK_ARCHON_TTM_TARGET_MIB must be set}"
: "${FRANK_ARCHON_RAD_MIN_MIB:?FRANK_ARCHON_RAD_MIN_MIB must be set}"
# The image gate is the digest-pinned ref itself: a name@sha256 ref exists
# in storage only if THAT manifest was pulled — a retagged look-alike or a
# newer latest does not satisfy it. `podman image exists` takes a
# name@digest ref in one command — no pipefail/grep race (the 2026-09-30
# lesson). Runs as dank: the unit's process uses dank's ROOTLESS podman
# storage (root unit, User=dank — the fleet's frank-archon shape).
if ! podman image exists "$FRANK_ARCHON_IMAGE"; then
echo "ERROR: radiance image absent: $FRANK_ARCHON_IMAGE" >&2
echo "frank-archon: podman pull docker://stilldeadcode/radiance:latest (as dank)" >&2
exit 1
fi
# The payload gate: the .rad container file, sized. A truncated convert
# would otherwise buy a guaranteed long-timeout boot instead of this fast
# failure. The staging dir / raw checkpoint are NOT this script's concern
# (scripts/build-frank-archon-rad.sh built the file; its rad-info output
# is the record).
RAD="$FRANK_ARCHON_MODEL_DIR/$FRANK_ARCHON_MODEL_FILE"
if [ ! -f "$RAD" ]; then
echo "ERROR: .rad container missing: $RAD" >&2
echo "frank-archon: run scripts/build-frank-archon-rad.sh" >&2
exit 1
fi
RAD_MIB=$(( $(stat -c %s "$RAD") / 1048576 ))
if [ "$RAD_MIB" -lt "$FRANK_ARCHON_RAD_MIN_MIB" ]; then
echo "ERROR: .rad container too small: $RAD is ${RAD_MIB} MiB, want >= $FRANK_ARCHON_RAD_MIN_MIB" >&2
echo "frank-archon: the convert was truncated — rebuild it" >&2
exit 1
fi
if [ ! -f "$FRANK_ARCHON_TEMPLATE" ]; then
echo "ERROR: chat template missing: $FRANK_ARCHON_TEMPLATE" >&2
exit 1
fi
mkdir -p "$FRANK_ARCHON_STATE_DIR/kvcache/frank-archon"
# The engine's create is NOT recursive: with only the state dir mounted, its
# "cannot create /data/kvcache/frank-archon: No such file or directory"
# left the SSD prefix tier off (measured 2026-10-06 boot ledger). Pre-create
# the full path on the host so the tier opens on first boot.
mkdir -p "$FRANK_ARCHON_STATE_DIR/tune"
# rad-tune's kernel-selection cache (see the runbook's "Tuning levers"):
# the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune at boot;
# $RADIANCE_HOME defaults to /opt/radiance, which lives in the image, so
# the cache must ride a host mount or it dies with the --rm container
# that wrote it. The dir is pre-created for the same reason as kvcache
# (the engine's create is not recursive).
if [ ! -e /dev/kfd ] || [ ! -d /dev/dri ]; then
echo "ERROR: GPU device nodes absent (/dev/kfd, /dev/dri)" >&2
exit 1
fi
# TTM preflight: the pinned ledger (2 x FRANK_ARCHON_HOST_POOL_MIB + the
# 47.7 GiB PLE table + the prefix host tier, ~147 GiB) rides TTM's shared
# pages, capped by ttm's pages_limit (the current kernel exposes NO
# pages_max; the cap function takes the min over the knobs that exist).
# Below FRANK_ARCHON_TTM_MIN_MIB the load dies mid-pin with an allocator
# error, so the gate refuses first — and RAISES the knob before refusing:
# the cap is a runtime knob whose default is half of RAM, so a power failure
# resets it (measured 2026-10-12 23:17 on this host: 200 GiB -> 125,988 MiB,
# and Restart=always re-ran the refusal 6,919 times over ~20 h).
# FRANK_ARCHON_TTM_TARGET_MIB is the recorded host ceiling, not the floor.
# shellcheck source=/dev/null
source "$SCRIPT_DIR/ttm-preflight.sh"
ttm_ensure_cap || exit 1
# A crashed/killed unit can leave the container name registered (systemd
# SIGKILLs conmon before cleanup runs). Clear OUR name plus the legacy
# identities (the FreeToken-era frank-archon-qwen, the retired R9V
# container and the ad hoc experiment containers all hold this GPU); all
# idempotent.
for CTR in $FRANK_ARCHON_LEGACY_CONTAINERS "$FRANK_ARCHON_CONTAINER"; do
podman rm -f "$CTR" >/dev/null 2>&1 || true
done
LOG=/home/dank/logs/frank_archon.log
mkdir -p "$(dirname "$LOG")"
# Publisher-vetted container contract (docs/DOCKER.md, compose/common.yaml):
# --network host (the server binds the host's addresses directly, no NAT),
# --init (the engine releases the cards on SIGTERM), seccomp=unconfined
# (Docker/podman's default profile refuses io_uring, which the n-gram
# table's reads go through, and the NUMA binding calls the ROCm runtime
# makes for host memory), memlock unbounded (the pinned ledger — legal
# because the root unit grants LimitMEMLOCK=infinity to this dank process),
# /dev/kfd + /dev/dri (render/video groups are moot for a root container).
# The template rides a single-file mount; the .rad dir is ro; /data holds
# the prefix-cache disk tier.
podman run -d --name "$FRANK_ARCHON_CONTAINER" \
--network host \
--init \
--security-opt seccomp=unconfined \
--ulimit memlock=-1:-1 \
--device /dev/kfd --device /dev/dri \
-v "$FRANK_ARCHON_MODEL_DIR":/models:ro \
-v "$FRANK_ARCHON_STATE_DIR":/data \
-v "$FRANK_ARCHON_STATE_DIR/tune":/opt/radiance/share/radiance/tune \
-v "$FRANK_ARCHON_TEMPLATE":/templates/template.jinja:ro \
"$FRANK_ARCHON_IMAGE" \
--model "/models/$FRANK_ARCHON_MODEL_FILE" \
--host :: --port "$FRANK_ARCHON_PORT" \
--served-model-name "$FRANK_ARCHON_SERVED_NAME" \
--override-chat-template /templates/template.jinja \
--tp "$FRANK_ARCHON_TP" --tp-wire "$FRANK_ARCHON_TP_WIRE" \
--max-num-seqs "$FRANK_ARCHON_MAX_SEQS" \
--max-model-len "$FRANK_ARCHON_MAX_MODEL_LEN" \
--placement "$FRANK_ARCHON_PLACEMENT" \
--host-pool-mib "$FRANK_ARCHON_HOST_POOL_MIB" \
--gpu-headroom-mib "$FRANK_ARCHON_GPU_HEADROOM_MIB" \
--vram-weights-mib "$FRANK_ARCHON_VRAM_WEIGHTS_MIB" \
--vram-kv-mib "$FRANK_ARCHON_VRAM_KV_MIB" \
--kv-cache-dtype "$FRANK_ARCHON_KV_DTYPE" \
--prefix-cache-host-mib "$FRANK_ARCHON_PREFIX_HOST_MIB" \
--prefix-cache-dir /data/kvcache/frank-archon \
--prefix-cache-disk-mib "$FRANK_ARCHON_PREFIX_DISK_MIB" \
--ngram-placement "$FRANK_ARCHON_NGRAM_PLACEMENT" \
--num-speculative-tokens "$FRANK_ARCHON_SPEC_TOKENS" \
--max-num-batched-tokens "$FRANK_ARCHON_MAX_BATCHED_TOKENS" \
${FRANK_ARCHON_EXTRA_ARGS:+$FRANK_ARCHON_EXTRA_ARGS} \
> "$LOG" 2>&1
# From here on, this script's lifetime IS the container's: wait returns the
# container's exit code, so a crash is a nonzero ExecStart and the unit's
# Restart=always is real. (systemd stop goes through ExecStop in the unit —
# podman wait alone cannot stop the container child.)
exec podman wait "$FRANK_ARCHON_CONTAINER"
@@ -1,13 +1,13 @@
#!/usr/bin/env bash
# serve-frank-large.sh — frank-large endpoint: halogen-flash-server 0.16.2
# serve-frank-dragoon.sh — frank-dragoon endpoint: halogen-flash-server 0.16.2
# (closed-source engine, image ghcr.io/peonist-ai/halogen-flash-server,
# digest-pinned in frank-large.env) serving the Quat3rnion abliterated
# digest-pinned in frank-dragoon.env) serving the Quat3rnion abliterated
# Swift1.5 finetune of peonist's Qwen3.8-Flash-Next v2 checkpoint on
# Dankclaw's Strix Halo iGPU, host :7770. 2026-10-06 cutover from the
# halogen 27B-dense shape — revert the swap commit's trio to roll back
# (the 27B payload stays on disk).
#
# The engine is driven by `podman run` DIRECTLY (the fleet's frank-primary
# The engine is driven by `podman run` DIRECTLY (the fleet's frank-archon
# pattern): the image starts engine + OpenAI frontend in one container.
# The frontend listens on 8731 inside (HALOGEN_API_PORT default) and is
# published as host 7770; the engine's own token port (8730, no auth)
@@ -26,13 +26,13 @@
# Why containers instead of a bare binary: the payload is .hgn and only
# the flash-server line reads it; it ships as a container; the root unit
# keeps LimitMEMLOCK=infinity for parity and to outlive the login session,
# exactly like frank-primary's.
# exactly like frank-archon's.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-large.env}"
ENV_FILE="${FRANK_DRAGOON_ENV_FILE:-$SCRIPT_DIR/frank-dragoon.env}"
if [ ! -f "$ENV_FILE" ]; then
echo "frank-large: env file not found: $ENV_FILE" >&2
echo "frank-dragoon: env file not found: $ENV_FILE" >&2
exit 1
fi
set -a
@@ -40,17 +40,17 @@ set -a
source "$ENV_FILE"
set +a
: "${FRANK_LARGE_IMAGE:?FRANK_LARGE_IMAGE must name the digest-pinned image}"
: "${FRANK_LARGE_CONTAINER:?FRANK_LARGE_CONTAINER must name the container}"
: "${FRANK_LARGE_PUBLISH:?FRANK_LARGE_PUBLISH must be host:container}"
: "${FRANK_LARGE_MODEL_DIR:?FRANK_LARGE_MODEL_DIR must name the payload dir}"
: "${FRANK_LARGE_CHECKPOINT:?FRANK_LARGE_CHECKPOINT must name the .hgn inside it}"
: "${FRANK_LARGE_CHECKPOINT_BYTES:?FRANK_LARGE_CHECKPOINT_BYTES must pin the size}"
: "${FRANK_LARGE_NGRAM_TABLE:?FRANK_LARGE_NGRAM_TABLE must name the lookup table}"
: "${FRANK_LARGE_NGRAM_BYTES:?FRANK_LARGE_NGRAM_BYTES must pin the table size}"
: "${FRANK_LARGE_VISION_TOWER:?FRANK_LARGE_VISION_TOWER must name the tower}"
: "${FRANK_LARGE_VISION_BYTES:?FRANK_LARGE_VISION_BYTES must pin the tower size}"
: "${FRANK_LARGE_TOKENIZER_FILES:?FRANK_LARGE_TOKENIZER_FILES must list the tokenizer dir contents}"
: "${FRANK_DRAGOON_IMAGE:?FRANK_DRAGOON_IMAGE must name the digest-pinned image}"
: "${FRANK_DRAGOON_CONTAINER:?FRANK_DRAGOON_CONTAINER must name the container}"
: "${FRANK_DRAGOON_PUBLISH:?FRANK_DRAGOON_PUBLISH must be host:container}"
: "${FRANK_DRAGOON_MODEL_DIR:?FRANK_DRAGOON_MODEL_DIR must name the payload dir}"
: "${FRANK_DRAGOON_CHECKPOINT:?FRANK_DRAGOON_CHECKPOINT must name the .hgn inside it}"
: "${FRANK_DRAGOON_CHECKPOINT_BYTES:?FRANK_DRAGOON_CHECKPOINT_BYTES must pin the size}"
: "${FRANK_DRAGOON_NGRAM_TABLE:?FRANK_DRAGOON_NGRAM_TABLE must name the lookup table}"
: "${FRANK_DRAGOON_NGRAM_BYTES:?FRANK_DRAGOON_NGRAM_BYTES must pin the table size}"
: "${FRANK_DRAGOON_VISION_TOWER:?FRANK_DRAGOON_VISION_TOWER must name the tower}"
: "${FRANK_DRAGOON_VISION_BYTES:?FRANK_DRAGOON_VISION_BYTES must pin the tower size}"
: "${FRANK_DRAGOON_TOKENIZER_FILES:?FRANK_DRAGOON_TOKENIZER_FILES must list the tokenizer dir contents}"
: "${HALOGEN_MODEL_ID:?HALOGEN_MODEL_ID must be the served alias (frank-dragoon)}"
: "${HALOGEN_CTX:?HALOGEN_CTX must be set (per-slot window)}"
: "${HALOGEN_KV_POOL_POSITIONS:?HALOGEN_KV_POOL_POSITIONS must be set}"
@@ -62,9 +62,9 @@ set +a
# podman dies SIGPIPE 141, pipefail turns the 141 into "no match" — the
# 2026-09-30 deploy refusal).
DIGESTS="$(podman images --format '{{.RepoDigests}}')"
if ! grep -qF "${FRANK_LARGE_IMAGE#*@}" <<<"$DIGESTS"; then
echo "frank-large: image $FRANK_LARGE_IMAGE not in podman storage;" >&2
echo " pull it: podman pull ${FRANK_LARGE_IMAGE%@*}" >&2
if ! grep -qF "${FRANK_DRAGOON_IMAGE#*@}" <<<"$DIGESTS"; then
echo "frank-dragoon: image $FRANK_DRAGOON_IMAGE not in podman storage;" >&2
echo " pull it: podman pull ${FRANK_DRAGOON_IMAGE%@*}" >&2
exit 1
fi
@@ -74,45 +74,45 @@ fi
ck() {
# ck <env-name> <path> <expected-bytes>
if [ ! -f "$2" ]; then
echo "frank-large: $1 $2 missing;" >&2
echo " fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
echo "frank-dragoon: $1 $2 missing;" >&2
echo " fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
exit 1
fi
local sz
sz="$(stat -c %s "$2")"
if [ "$sz" != "$3" ]; then
echo "frank-large: $1 is $sz bytes, expected $3 (partial download?)" >&2
echo "frank-dragoon: $1 is $sz bytes, expected $3 (partial download?)" >&2
exit 1
fi
}
ck checkpoint "$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_CHECKPOINT" "$FRANK_LARGE_CHECKPOINT_BYTES"
ck n-gram-table "$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_NGRAM_TABLE" "$FRANK_LARGE_NGRAM_BYTES"
ck vision-tower "$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_VISION_TOWER" "$FRANK_LARGE_VISION_BYTES"
ck checkpoint "$FRANK_DRAGOON_MODEL_DIR/$FRANK_DRAGOON_CHECKPOINT" "$FRANK_DRAGOON_CHECKPOINT_BYTES"
ck n-gram-table "$FRANK_DRAGOON_MODEL_DIR/$FRANK_DRAGOON_NGRAM_TABLE" "$FRANK_DRAGOON_NGRAM_BYTES"
ck vision-tower "$FRANK_DRAGOON_MODEL_DIR/$FRANK_DRAGOON_VISION_TOWER" "$FRANK_DRAGOON_VISION_BYTES"
# The payload ships the tokenizer as a FLAT DIRECTORY. A HuggingFace cache
# snapshot symlinks each file into a sibling blobs/, and those symlinks
# dangle inside the container mount — so the directory must be real and
# every named file must be a regular file, not a link.
TOK_DIR="$FRANK_LARGE_MODEL_DIR/tokenizer"
TOK_DIR="$FRANK_DRAGOON_MODEL_DIR/tokenizer"
if [ -L "$TOK_DIR" ] || [ ! -d "$TOK_DIR" ]; then
echo "frank-large: tokenizer dir $TOK_DIR missing or a symlink (HF cache snapshot?);" >&2
echo " fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
echo "frank-dragoon: tokenizer dir $TOK_DIR missing or a symlink (HF cache snapshot?);" >&2
echo " fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
exit 1
fi
for TF in $FRANK_LARGE_TOKENIZER_FILES; do
for TF in $FRANK_DRAGOON_TOKENIZER_FILES; do
if [ ! -f "$TOK_DIR/$TF" ] || [ -L "$TOK_DIR/$TF" ]; then
echo "frank-large: tokenizer file $TOK_DIR/$TF missing or a symlink;" >&2
echo " fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
echo "frank-dragoon: tokenizer file $TOK_DIR/$TF missing or a symlink;" >&2
echo " fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
exit 1
fi
done
# GPU access is mandatory but the node paths are variables so the seam can
# drive the gate on a box without an AMD GPU.
DEV_KFD="${FRANK_LARGE_DEV_KFD:-/dev/kfd}"
DEV_DRI="${FRANK_LARGE_DEV_DRI:-/dev/dri}"
DEV_KFD="${FRANK_DRAGOON_DEV_KFD:-/dev/kfd}"
DEV_DRI="${FRANK_DRAGOON_DEV_DRI:-/dev/dri}"
if [ ! -e "$DEV_KFD" ] || [ ! -d "$DEV_DRI" ]; then
echo "frank-large: $DEV_KFD or $DEV_DRI missing — no gfx1151 to serve on" >&2
echo "frank-dragoon: $DEV_KFD or $DEV_DRI missing — no gfx1151 to serve on" >&2
exit 1
fi
# Host-RAM preflight — the engine's pin gate, checked BEFORE podman run.
@@ -125,15 +125,15 @@ fi
# self-resolved in ~2 min when the dying 27B engine freed RAM. Check here and
# WAIT for the RAM to clear, so a transient dip boots once instead of
# crash-looping; if it never clears, refuse loudly.
MEMINFO="${FRANK_LARGE_MEMINFO:-/proc/meminfo}"
PIN_KB=$(( FRANK_LARGE_CHECKPOINT_BYTES / 1024 ))
MEMINFO="${FRANK_DRAGOON_MEMINFO:-/proc/meminfo}"
PIN_KB=$(( FRANK_DRAGOON_CHECKPOINT_BYTES / 1024 ))
NEED_KB=$(( PIN_KB + 16 * 1048576 ))
ram_avail_kb() { awk '/^MemAvailable:/{print $2; exit}' "$MEMINFO"; }
MEMAVAIL_KB="$(ram_avail_kb)" || MEMAVAIL_KB=""
if [ -n "$MEMAVAIL_KB" ] && [ "$MEMAVAIL_KB" -lt "$NEED_KB" ]; then
WAIT_S="${FRANK_LARGE_RAM_WAIT:-600}"
WAIT_S="${FRANK_DRAGOON_RAM_WAIT:-600}"
GIB() { awk -v kb="$1" 'BEGIN{printf "%.2f", kb/1048576}'; }
echo "frank-large: host RAM below the pin gate — MemAvailable $(GIB "$MEMAVAIL_KB") GiB < needed $(GIB "$NEED_KB") GiB ($(GIB "$PIN_KB") GiB checkpoint + 16.00 GiB floor); waiting up to ${WAIT_S}s for host RAM to clear" >&2
echo "frank-dragoon: host RAM below the pin gate — MemAvailable $(GIB "$MEMAVAIL_KB") GiB < needed $(GIB "$NEED_KB") GiB ($(GIB "$PIN_KB") GiB checkpoint + 16.00 GiB floor); waiting up to ${WAIT_S}s for host RAM to clear" >&2
deadline=$(( $(date +%s) + WAIT_S ))
while [ "$(date +%s)" -lt "$deadline" ]; do
sleep 5
@@ -143,16 +143,16 @@ if [ -n "$MEMAVAIL_KB" ] && [ "$MEMAVAIL_KB" -lt "$NEED_KB" ]; then
fi
done
if [ -z "$MEMAVAIL_KB" ] || [ "$MEMAVAIL_KB" -lt "$NEED_KB" ]; then
echo "frank-large: MemAvailable $(GIB "${MEMAVAIL_KB:-0}") GiB after ${WAIT_S}s; refusing to start (the engine would refuse the pin)" >&2
echo "frank-dragoon: MemAvailable $(GIB "${MEMAVAIL_KB:-0}") GiB after ${WAIT_S}s; refusing to start (the engine would refuse the pin)" >&2
exit 1
fi
echo "frank-large: MemAvailable cleared to $(GIB "$MEMAVAIL_KB") GiB; starting" >&2
echo "frank-dragoon: MemAvailable cleared to $(GIB "$MEMAVAIL_KB") GiB; starting" >&2
fi
for CTR in ${FRANK_LARGE_LEGACY_CONTAINERS:-} "$FRANK_LARGE_CONTAINER"; do
for CTR in ${FRANK_DRAGOON_LEGACY_CONTAINERS:-} "$FRANK_DRAGOON_CONTAINER"; do
podman rm -f "$CTR" >/dev/null 2>&1 || true
done
LOG="${FRANK_LARGE_LOG:-/home/dank/logs/frank_large.log}"
LOG="${FRANK_DRAGOON_LOG:-/home/dank/logs/frank_dragoon.log}"
mkdir -p "$(dirname "$LOG")"
# --ipc=host for the udev-granted supplementary group, seccomp unconfined
# for the ROCm ioctl surface, and unbounded memlock (the unit's
@@ -163,17 +163,17 @@ mkdir -p "$(dirname "$LOG")"
# and both sibling serve scripts pass no nofile either. No comment lines
# inside the backslash chain — a mid-chain # terminates the continuation
# (the 2026-10-05 exit-127 crash loop on the sibling serve script).
podman run --name "$FRANK_LARGE_CONTAINER" \
--publish "$FRANK_LARGE_PUBLISH" \
podman run --name "$FRANK_DRAGOON_CONTAINER" \
--publish "$FRANK_DRAGOON_PUBLISH" \
--ipc=host \
--ulimit memlock=-1:-1 \
--device "$DEV_KFD" --device "$DEV_DRI" \
--group-add keep-groups \
--security-opt seccomp=unconfined \
-e HALOGEN_MODEL_ID="$HALOGEN_MODEL_ID" \
-e HALOGEN_CHECKPOINT="/models/$FRANK_LARGE_CHECKPOINT" \
-e HALOGEN_NGRAM_TABLE="/models/$FRANK_LARGE_NGRAM_TABLE" \
-e HALOGEN_VISION_TOWER="/models/$FRANK_LARGE_VISION_TOWER" \
-e HALOGEN_CHECKPOINT="/models/$FRANK_DRAGOON_CHECKPOINT" \
-e HALOGEN_NGRAM_TABLE="/models/$FRANK_DRAGOON_NGRAM_TABLE" \
-e HALOGEN_VISION_TOWER="/models/$FRANK_DRAGOON_VISION_TOWER" \
-e HALOGEN_TOKENIZER=/tokenizer \
-e HALOGEN_CTX="$HALOGEN_CTX" \
-e HALOGEN_KV_POOL_POSITIONS="$HALOGEN_KV_POOL_POSITIONS" \
@@ -187,9 +187,9 @@ podman run --name "$FRANK_LARGE_CONTAINER" \
-e HALOGEN_TOP_P="${HALOGEN_TOP_P:-0.95}" \
-e HALOGEN_TOP_K="${HALOGEN_TOP_K:-20}" \
${HALOGEN_WEIGHTS_LOCK:+-e HALOGEN_WEIGHTS_LOCK="$HALOGEN_WEIGHTS_LOCK"} \
-v "$FRANK_LARGE_MODEL_DIR:/models:ro" \
-v "$FRANK_LARGE_MODEL_DIR/tokenizer:/tokenizer:ro" \
"$FRANK_LARGE_IMAGE" \
-v "$FRANK_DRAGOON_MODEL_DIR:/models:ro" \
-v "$FRANK_DRAGOON_MODEL_DIR/tokenizer:/tokenizer:ro" \
"$FRANK_DRAGOON_IMAGE" \
> "$LOG" 2>&1
# FOREGROUND `podman run`: the engine's stdout/stderr are this script's
-206
View File
@@ -1,206 +0,0 @@
#!/usr/bin/env bash
# serve-frank-primary.sh — frank-primary endpoint: radiance (StillDeadcode,
# Apache-2.0) serving the OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint at
# NATIVE block-FP8 precision from a rad-convert container
# (/home/dank/models/rad/qwen4exp-fp8-native.rad), on BOTH R9700s (TP=2,
# wht6 wire), dankserver :7770. 2026-10-05 cutover from FreeToken (PR #587,
# one card; git tag frank-primary-freetoken-pr587-20261005 is the
# pre-cutover state).
#
# The run line below IS the engine argv: the image's entrypoint is the
# engine and its tools sit beside it on PATH. Unlike every predecessor
# there is no per-boot VRAM planner to second-guess: the boot ledger in the
# log states every budget before anything is allocated (weights vs KV
# split, the expert host pool's coverage of the worst case, the PLE pin,
# the P2P answer per pair) — read it after every resize. First-boot shape:
# 2 decode slots (Frank's call), 40 GiB/rank pinned expert pool, fp8 KV,
# MTP depth 3, prefix cache host 4 GiB + disk 64 GiB, PLE pinned in RAM.
#
# The root unit (scripts/frank-primary.service, User=dank,
# LimitMEMLOCK=infinity) grants the pin budget: 2 x 40 GiB expert pools +
# the 47.7 GiB PLE table + 4 GiB prefix host tier — ~132 GiB of pinned
# pages, which is ALSO why this script refuses to start when ttm's
# pages_limit/pages_max cap is under FRANK_PRIMARY_TTM_MIN_MIB (see below).
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${FRANK_PRIMARY_ENV_FILE:-$SCRIPT_DIR/frank-primary.env}"
if [ ! -f "$ENV_FILE" ]; then
echo "ERROR: frank-primary env file absent: $ENV_FILE" >&2
exit 1
fi
set -a
# shellcheck disable=SC1090
source "$ENV_FILE"
set +a
: "${FRANK_PRIMARY_IMAGE:?FRANK_PRIMARY_IMAGE must name the radiance image}"
: "${FRANK_PRIMARY_MODEL_DIR:?FRANK_PRIMARY_MODEL_DIR must name the .rad dir}"
: "${FRANK_PRIMARY_MODEL_FILE:?FRANK_PRIMARY_MODEL_FILE must name the .rad file}"
: "${FRANK_PRIMARY_STATE_DIR:?FRANK_PRIMARY_STATE_DIR must name the /data state dir}"
: "${FRANK_PRIMARY_CONTAINER:?FRANK_PRIMARY_CONTAINER must name the container}"
: "${FRANK_PRIMARY_SERVED_NAME:?FRANK_PRIMARY_SERVED_NAME must name the alias}"
: "${FRANK_PRIMARY_PORT:?FRANK_PRIMARY_PORT must be the port}"
: "${FRANK_PRIMARY_TEMPLATE:?FRANK_PRIMARY_TEMPLATE must name the chat template}"
: "${FRANK_PRIMARY_TP:?FRANK_PRIMARY_TP must be set}"
: "${FRANK_PRIMARY_TP_WIRE:?FRANK_PRIMARY_TP_WIRE must be set}"
: "${FRANK_PRIMARY_MAX_SEQS:?FRANK_PRIMARY_MAX_SEQS must be set}"
: "${FRANK_PRIMARY_MAX_MODEL_LEN:?FRANK_PRIMARY_MAX_MODEL_LEN must be set}"
: "${FRANK_PRIMARY_PLACEMENT:?FRANK_PRIMARY_PLACEMENT must be set}"
: "${FRANK_PRIMARY_HOST_POOL_MIB:?FRANK_PRIMARY_HOST_POOL_MIB must be set}"
: "${FRANK_PRIMARY_GPU_HEADROOM_MIB:?FRANK_PRIMARY_GPU_HEADROOM_MIB must be set}"
: "${FRANK_PRIMARY_VRAM_WEIGHTS_MIB:?FRANK_PRIMARY_VRAM_WEIGHTS_MIB must be set}"
: "${FRANK_PRIMARY_VRAM_KV_MIB:?FRANK_PRIMARY_VRAM_KV_MIB must be set}"
: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set}"
: "${FRANK_PRIMARY_SPEC_TOKENS:?FRANK_PRIMARY_SPEC_TOKENS must be set}"
: "${FRANK_PRIMARY_MAX_BATCHED_TOKENS:?FRANK_PRIMARY_MAX_BATCHED_TOKENS must be set}"
: "${FRANK_PRIMARY_PREFIX_HOST_MIB:?FRANK_PRIMARY_PREFIX_HOST_MIB must be set}"
: "${FRANK_PRIMARY_PREFIX_DISK_MIB:?FRANK_PRIMARY_PREFIX_DISK_MIB must be set}"
: "${FRANK_PRIMARY_NGRAM_PLACEMENT:?FRANK_PRIMARY_NGRAM_PLACEMENT must be set}"
: "${FRANK_PRIMARY_TTM_MIN_MIB:?FRANK_PRIMARY_TTM_MIN_MIB must be set}"
: "${FRANK_PRIMARY_RAD_MIN_MIB:?FRANK_PRIMARY_RAD_MIN_MIB must be set}"
# The image gate is the digest-pinned ref itself: a name@sha256 ref exists
# in storage only if THAT manifest was pulled — a retagged look-alike or a
# newer latest does not satisfy it. `podman image exists` takes a
# name@digest ref in one command — no pipefail/grep race (the 2026-09-30
# lesson). Runs as dank: the unit's process uses dank's ROOTLESS podman
# storage (root unit, User=dank — the fleet's frank-primary shape).
if ! podman image exists "$FRANK_PRIMARY_IMAGE"; then
echo "ERROR: radiance image absent: $FRANK_PRIMARY_IMAGE" >&2
echo "frank-primary: podman pull docker://stilldeadcode/radiance:latest (as dank)" >&2
exit 1
fi
# The payload gate: the .rad container file, sized. A truncated convert
# would otherwise buy a guaranteed long-timeout boot instead of this fast
# failure. The staging dir / raw checkpoint are NOT this script's concern
# (scripts/build-frank-primary-rad.sh built the file; its rad-info output
# is the record).
RAD="$FRANK_PRIMARY_MODEL_DIR/$FRANK_PRIMARY_MODEL_FILE"
if [ ! -f "$RAD" ]; then
echo "ERROR: .rad container missing: $RAD" >&2
echo "frank-primary: run scripts/build-frank-primary-rad.sh" >&2
exit 1
fi
RAD_MIB=$(( $(stat -c %s "$RAD") / 1048576 ))
if [ "$RAD_MIB" -lt "$FRANK_PRIMARY_RAD_MIN_MIB" ]; then
echo "ERROR: .rad container too small: $RAD is ${RAD_MIB} MiB, want >= $FRANK_PRIMARY_RAD_MIN_MIB" >&2
echo "frank-primary: the convert was truncated — rebuild it" >&2
exit 1
fi
if [ ! -f "$FRANK_PRIMARY_TEMPLATE" ]; then
echo "ERROR: chat template missing: $FRANK_PRIMARY_TEMPLATE" >&2
exit 1
fi
mkdir -p "$FRANK_PRIMARY_STATE_DIR/kvcache/frank-primary"
# The engine's create is NOT recursive: with only the state dir mounted, its
# "cannot create /data/kvcache/frank-primary: No such file or directory"
# left the SSD prefix tier off (measured 2026-10-06 boot ledger). Pre-create
# the full path on the host so the tier opens on first boot.
mkdir -p "$FRANK_PRIMARY_STATE_DIR/tune"
# rad-tune's kernel-selection cache (see the runbook's "Tuning levers"):
# the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune at boot;
# $RADIANCE_HOME defaults to /opt/radiance, which lives in the image, so
# the cache must ride a host mount or it dies with the --rm container
# that wrote it. The dir is pre-created for the same reason as kvcache
# (the engine's create is not recursive).
if [ ! -e /dev/kfd ] || [ ! -d /dev/dri ]; then
echo "ERROR: GPU device nodes absent (/dev/kfd, /dev/dri)" >&2
exit 1
fi
# TTM preflight: the pinned ledger (~2 x 40 GiB pools + 47.7 GiB PLE + 4 GiB
# prefix host = ~132 GiB) rides TTM's shared pages, capped by ttm's
# pages_limit (the current kernel exposes NO pages_max; the cap function
# takes the min over the knobs that exist). Below FRANK_PRIMARY_TTM_MIN_MIB
# the load dies mid-pin with an allocator error instead of this named
# refusal — raise the knob first (runtime sysfs write; kernel cmdline
# fallback + one reboot only if the file is read-only), see the runbook.
# RAISED 2026-10-05 pre-cutover to 52,428,800 pages = 200 GiB.
ttm_cap_mib() {
# min over the knobs THIS kernel exposes — the current kernel has
# pages_limit only (no pages_max); a knob that is absent is skipped,
# not a zero (a zero would refuse a healthy cap).
local v cur=""
for p in /sys/module/ttm/parameters/pages_limit /sys/module/ttm/parameters/pages_max; do
[ -r "$p" ] || continue
v="$(cat "$p" 2>/dev/null)" || continue
[ -n "$v" ] || continue
v=$(( v * 4 / 1024 ))
if [ "$v" -lt "${cur:-999999999}" ]; then cur=$v; fi
done
echo "${cur:-0}"
}
TTM_CAP="$(ttm_cap_mib)"
if [ "$TTM_CAP" -lt "$FRANK_PRIMARY_TTM_MIN_MIB" ]; then
echo "ERROR: ttm pages cap is ${TTM_CAP} MiB, the pinned ledger needs >= $FRANK_PRIMARY_TTM_MIN_MIB" >&2
echo "frank-primary: raise /sys/module/ttm/parameters/pages_limit (and pages_max if the kernel exposes one) to >= $FRANK_PRIMARY_TTM_MIN_MIB in 4 KiB pages" >&2
exit 1
fi
# A crashed/killed unit can leave the container name registered (systemd
# SIGKILLs conmon before cleanup runs). Clear OUR name plus the legacy
# identities (the FreeToken-era frank-primary-qwen, the retired R9V
# container and the ad hoc experiment containers all hold this GPU); all
# idempotent.
for CTR in $FRANK_PRIMARY_LEGACY_CONTAINERS "$FRANK_PRIMARY_CONTAINER"; do
podman rm -f "$CTR" >/dev/null 2>&1 || true
done
LOG=/home/dank/logs/frank_primary.log
mkdir -p "$(dirname "$LOG")"
# Publisher-vetted container contract (docs/DOCKER.md, compose/common.yaml):
# --network host (the server binds the host's addresses directly, no NAT),
# --init (the engine releases the cards on SIGTERM), seccomp=unconfined
# (Docker/podman's default profile refuses io_uring, which the n-gram
# table's reads go through, and the NUMA binding calls the ROCm runtime
# makes for host memory), memlock unbounded (the pinned ledger — legal
# because the root unit grants LimitMEMLOCK=infinity to this dank process),
# /dev/kfd + /dev/dri (render/video groups are moot for a root container).
# The template rides a single-file mount; the .rad dir is ro; /data holds
# the prefix-cache disk tier.
podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
--network host \
--init \
--security-opt seccomp=unconfined \
--ulimit memlock=-1:-1 \
--device /dev/kfd --device /dev/dri \
-v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
-v "$FRANK_PRIMARY_STATE_DIR":/data \
-v "$FRANK_PRIMARY_STATE_DIR/tune":/opt/radiance/share/radiance/tune \
-v "$FRANK_PRIMARY_TEMPLATE":/templates/template.jinja:ro \
"$FRANK_PRIMARY_IMAGE" \
--model "/models/$FRANK_PRIMARY_MODEL_FILE" \
--host :: --port "$FRANK_PRIMARY_PORT" \
--served-model-name "$FRANK_PRIMARY_SERVED_NAME" \
--override-chat-template /templates/template.jinja \
--tp "$FRANK_PRIMARY_TP" --tp-wire "$FRANK_PRIMARY_TP_WIRE" \
--max-num-seqs "$FRANK_PRIMARY_MAX_SEQS" \
--max-model-len "$FRANK_PRIMARY_MAX_MODEL_LEN" \
--placement "$FRANK_PRIMARY_PLACEMENT" \
--host-pool-mib "$FRANK_PRIMARY_HOST_POOL_MIB" \
--gpu-headroom-mib "$FRANK_PRIMARY_GPU_HEADROOM_MIB" \
--vram-weights-mib "$FRANK_PRIMARY_VRAM_WEIGHTS_MIB" \
--vram-kv-mib "$FRANK_PRIMARY_VRAM_KV_MIB" \
--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
--prefix-cache-host-mib "$FRANK_PRIMARY_PREFIX_HOST_MIB" \
--prefix-cache-dir /data/kvcache/frank-primary \
--prefix-cache-disk-mib "$FRANK_PRIMARY_PREFIX_DISK_MIB" \
--ngram-placement "$FRANK_PRIMARY_NGRAM_PLACEMENT" \
--num-speculative-tokens "$FRANK_PRIMARY_SPEC_TOKENS" \
--max-num-batched-tokens "$FRANK_PRIMARY_MAX_BATCHED_TOKENS" \
${FRANK_PRIMARY_EXTRA_ARGS:+$FRANK_PRIMARY_EXTRA_ARGS} \
> "$LOG" 2>&1
# From here on, this script's lifetime IS the container's: wait returns the
# container's exit code, so a crash is a nonzero ExecStart and the unit's
# Restart=always is real. (systemd stop goes through ExecStop in the unit —
# podman wait alone cannot stop the container child.)
exec podman wait "$FRANK_PRIMARY_CONTAINER"
@@ -1,6 +1,6 @@
#!/usr/bin/env bash
# Serve HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive (Q6_K) as
# frank-secondary on Dankclaw's RTX 3090 via llama.cpp (master build, CUDA sm_86).
# frank-zealot on Dankclaw's RTX 3090 via llama.cpp (master build, CUDA sm_86).
#
# 2026-10-07 (Frank): Qwen3.5-9B swap at TWO 200,704-token conversations
# (--ctx-size 401408 --parallel 2) with EXACT f16 KV — the 9B rung frees the
@@ -24,11 +24,11 @@
# Sampling: llama.cpp takes it per request (the fleet harness sends the
# publisher recipe from models.yml: thinking temp 0.6 / top_p 0.95 / top_k 20).
#
# Tunables come from frank-secondary.env unless FRANK_SECONDARY_ENV_FILE overrides.
# Tunables come from frank-zealot.env unless FRANK_ZEALOT_ENV_FILE overrides.
set -euo pipefail
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${FRANK_SECONDARY_ENV_FILE:-$SCRIPT_DIR/frank-secondary.env}"
ENV_FILE="${FRANK_ZEALOT_ENV_FILE:-$SCRIPT_DIR/frank-zealot.env}"
if [[ -f "$ENV_FILE" ]]; then
set -a
# shellcheck disable=SC1090
@@ -83,11 +83,11 @@ fi
# has not finished, and Restart=always would turn it into a crash loop that
# reads like a runtime fault.
for path in "$LLAMA_BIN" "$MODEL_FILE"; do
[[ -e "$path" ]] || { echo "frank-secondary: missing $path" >&2; exit 1; }
[[ -e "$path" ]] || { echo "frank-zealot: missing $path" >&2; exit 1; }
done
EXTRA_ARGS=()
if [[ -n "$MMPROJ_FILE" ]]; then
[[ -e "$MMPROJ_FILE" ]] || { echo "frank-secondary: missing $MMPROJ_FILE" >&2; exit 1; }
[[ -e "$MMPROJ_FILE" ]] || { echo "frank-zealot: missing $MMPROJ_FILE" >&2; exit 1; }
EXTRA_ARGS+=(--mmproj "$MMPROJ_FILE")
fi
@@ -97,7 +97,7 @@ fi
if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then
EXTRA_ARGS+=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
if [[ -n "$SPEC_DRAFT_FILE" ]]; then
[[ -e "$SPEC_DRAFT_FILE" ]] || { echo "frank-secondary: missing $SPEC_DRAFT_FILE" >&2; exit 1; }
[[ -e "$SPEC_DRAFT_FILE" ]] || { echo "frank-zealot: missing $SPEC_DRAFT_FILE" >&2; exit 1; }
EXTRA_ARGS+=(--spec-draft-model "$SPEC_DRAFT_FILE")
fi
fi
@@ -108,7 +108,7 @@ nvidia-modprobe -u -c=0 2>/dev/null || true
# Flash attention stays pinned on: it is REQUIRED for quantized V (q8_0),
# and the build carries GGML_CUDA_FA_ALL_QUANTS=ON (since 2026-09-17), so
# the q8_0 KV contract in frank-secondary.env is a real path — without this
# the q8_0 KV contract in frank-zealot.env is a real path — without this
# flag or that build option the engine silently falls back and the
# contract becomes a lie.
ARGS=(
+11 -11
View File
@@ -145,12 +145,12 @@ for path in "${CHANGED[@]}"; do
add_test tests/homepage_caddy.bats
add_test tests/deployment_scoping.bats
;;
# frank-large is a rootless quadlet again since 2026-09-16 (the
# frank-dragoon is a rootless quadlet again since 2026-09-16 (the
# halogen-flash-server restore). Its unit carries the fleet's GPU and
# podman posture, so it gets its own seam rather than the generic
# fallback below.
dankclaw/frank-large/*)
add_test tests/frank_large_unit.bats
dankclaw/frank-dragoon/*)
add_test tests/frank_dragoon_unit.bats
add_test tests/deployment_scoping.bats
add_test tests/docs_completeness.bats
;;
@@ -187,29 +187,29 @@ for path in "${CHANGED[@]}"; do
# as additions, with "no focused-test mapping".
.AGENTTRASH/*|*/.AGENTTRASH/*)
;;
# frank-primary is llama.cpp (the unsloth b11030-mix prebuilt) on the
# frank-archon is llama.cpp (the unsloth b11030-mix prebuilt) on the
# two R9700s as a ROOT unit — the 2026-09-26 swap that replaced the
# Lucebox DS4.1 quadlet on :7770. Its env, unit and serve script all
# live in scripts/, which is why this case has to precede the generic
# scripts/*.service and scripts/* fallbacks below.
scripts/frank-primary.*|scripts/serve-frank-primary.sh)
add_test tests/frank_primary_unit.bats
scripts/frank-archon.*|scripts/serve-frank-archon.sh)
add_test tests/frank_archon_unit.bats
add_test tests/llm_units.bats
add_test tests/deployment_scoping.bats
add_test tests/scripts_syntax.bats
;;
# frank-large's chat template lives in scripts/ (moved there on
# frank-dragoon's chat template lives in scripts/ (moved there on
# 2026-09-26 when its old home was retired): a template edit changes
# what the live endpoint renders, so it selects the large unit seam.
scripts/qwen-sharp-v22.4.0.jinja)
add_test tests/frank_large_unit.bats
add_test tests/frank_dragoon_unit.bats
;;
# frank-secondary is llama.cpp on the 3090 as a ROOT unit: a native CUDA
# frank-zealot is llama.cpp on the 3090 as a ROOT unit: a native CUDA
# binary, so no generated quadlet is involved. Its env, unit and serve
# script all live in scripts/, which is why this case has to precede
# the generic scripts/*.service and scripts/* fallbacks below.
scripts/frank-secondary.*|scripts/serve-frank-secondary.sh)
add_test tests/frank_secondary_unit.bats
scripts/frank-zealot.*|scripts/serve-frank-zealot.sh)
add_test tests/frank_zealot_unit.bats
add_test tests/llm_units.bats
add_test tests/deployment_scoping.bats
add_test tests/scripts_syntax.bats
+104
View File
@@ -0,0 +1,104 @@
#!/usr/bin/env bash
# ttm-preflight.sh — the TTM pages-cap gate for engines that PIN host RAM.
#
# Sourced, never executed. An engine that pins tens of GiB of host RAM (the
# radiance expert pools + the PLE n-gram table) has its pinned pages charged
# against the kernel's ttm page cap: /sys/module/ttm/parameters/pages_limit
# (and pages_max on kernels that expose one). The cap function below takes the
# MIN over the knobs the running kernel actually exposes — an absent knob is
# skipped, never read as zero, because a zero would refuse a healthy cap.
#
# THE LAW THIS FILE EXISTS FOR: the cap is a RUNTIME knob. The kernel default is
# half of RAM, so every raise is lost on reboot. On 2026-10-12 23:17 a power
# failure on dankserver reset it 200 GiB -> 125,988 MiB, and frank-archon's
# preflight refused the start — 6,919 crash-restarts over ~20 hours, with
# `Restart=always` cheerfully re-running a refusal. A manual sysfs write is
# therefore not an acceptable repair procedure: the gate raises the knob itself
# (the units that need it run as root, and the knob is a 0644 root-owned module
# parameter), re-reads, and only then refuses.
#
# Contract (set by the sourcing script's env file):
# FRANK_ARCHON_TTM_MIN_MIB floor the pinned ledger needs; refuse below it
# FRANK_ARCHON_TTM_TARGET_MIB value to raise the cap to when it is below the
# floor (the recorded host ceiling — writing the
# floor itself would leave the ledger no slack)
# FRANK_ARCHON_TTM_SYS_DIR knob directory; defaults to the real sysfs
# path, a test points it at a temp dir.
# Pages are 4 KiB: pages = MiB x 256.
: "${FRANK_ARCHON_TTM_SYS_DIR:=/sys/module/ttm/parameters}"
# Prints the effective cap in MiB, 0 when no knob is readable.
ttm_cap_mib() {
local p v cur=""
for p in "$FRANK_ARCHON_TTM_SYS_DIR/pages_limit" \
"$FRANK_ARCHON_TTM_SYS_DIR/pages_max"; do
[ -r "$p" ] || continue
v="$(cat "$p" 2>/dev/null)" || continue
[ -n "$v" ] || continue
v=$(( v * 4 / 1024 ))
if [ "$v" -lt "${cur:-999999999}" ]; then cur=$v; fi
done
echo "${cur:-0}"
}
# Writes $1 MiB (as 4 KiB pages) into every knob present. Failures are
# swallowed: the caller re-reads the cap, and a write the kernel refused shows
# up there (ttm refuses a limit below the pages it has already pinned).
#
# PRIVILEGE, measured the hard way: the knob is root-owned 0644 and the service
# process runs as User=dank, so `-w` is FALSE there and a plain echo fails. The
# canonical privileged path is the unit's ExecStartPre with the `+` prefix
# (systemd runs it as root even under User=); `sudo -n` is the fallback for a
# human running the gate from a shell (passwordless sudo on both hosts).
ttm_raise_cap() {
local pages=$(( $1 * 256 )) p
for p in "$FRANK_ARCHON_TTM_SYS_DIR/pages_limit" \
"$FRANK_ARCHON_TTM_SYS_DIR/pages_max"; do
[ -e "$p" ] || continue
if [ -w "$p" ]; then
echo "$pages" > "$p" 2>/dev/null || true
elif command -v sudo >/dev/null 2>&1; then
sudo -n sh -c "echo $pages > '$p'" >/dev/null 2>&1 || true
fi
done
}
# Returns 0 when the cap clears FRANK_ARCHON_TTM_MIN_MIB, raising it first if
# needed; returns 1 with a named refusal when it cannot be made to.
ttm_ensure_cap() {
local cap
cap="$(ttm_cap_mib)"
if [ "$cap" -lt "$FRANK_ARCHON_TTM_MIN_MIB" ]; then
echo "frank-archon: ttm pages cap is ${cap} MiB, below the pinned-ledger" \
"floor $FRANK_ARCHON_TTM_MIN_MIB — raising the knob to" \
"$FRANK_ARCHON_TTM_TARGET_MIB MiB ($(( FRANK_ARCHON_TTM_TARGET_MIB * 256 )) pages)" >&2
ttm_raise_cap "$FRANK_ARCHON_TTM_TARGET_MIB"
cap="$(ttm_cap_mib)"
fi
if [ "$cap" -lt "$FRANK_ARCHON_TTM_MIN_MIB" ]; then
echo "ERROR: ttm pages cap is ${cap} MiB, the pinned ledger needs >= $FRANK_ARCHON_TTM_MIN_MIB" >&2
echo "frank-archon: raise $FRANK_ARCHON_TTM_SYS_DIR/pages_limit (and pages_max if the kernel exposes one) to >= $(( FRANK_ARCHON_TTM_MIN_MIB * 256 )) in 4 KiB pages; a read-only knob needs the module option (options ttm pages_limit=N) and a reboot" >&2
return 1
fi
echo "frank-archon: ttm pages cap ${cap} MiB clears the pinned-ledger floor $FRANK_ARCHON_TTM_MIN_MIB" >&2
}
# Executed (not sourced) it is the privileged pre-start hook: systemd runs this
# as root from the unit's `ExecStartPre=+…`, where the knob write is legal and
# the `Restart=always` loop can never outrun it.
if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
set -euo pipefail
case "${1:-}" in
--raise)
TTM_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=/dev/null
source "${FRANK_ARCHON_ENV_FILE:-$TTM_DIR/frank-archon.env}"
ttm_ensure_cap
;;
*)
echo "usage: ${0##*/} --raise (raise the ttm pages cap, then verify)" >&2
exit 2
;;
esac
fi
+10 -10
View File
@@ -16,13 +16,13 @@ this file is repo-managed; edits are harness-side only.
**Model ids are the StarCraft unit names (official 2026-10-08, on the wire
since 2026-10-09):** `frank-archon` (dankserver :7770, radiance),
`frank-dragoon` (dankclaw :7770, flash-server), `frank-zealot` (dankclaw
:7771, the 3090 compaction hub) — formerly `frank-primary`, `frank-large`,
`frank-secondary`. The SERVICES keep the old names (units, scripts, env
:7771, the 3090 compaction hub) — formerly `frank-archon`, `frank-dragoon`,
`frank-zealot`. The SERVICES keep the old names (units, scripts, env
files, runbook pages); pages written in the old era name the service,
the harness names the model. The wire now matches: each endpoint's
`/v1/models` answers with the Protoss name, so `models.yml`, the OpenWebUI
picker/grants, the homepage docs and the raw API all use one id per model
(see [[OpenWebUI]] for the grant re-key and the [[frank-primary-radiance-swift15-w4dn8]]
(see [[OpenWebUI]] for the grant re-key and the [[frank-archon-radiance-swift15-w4dn8]]
note on radiance echoing the requested name).
## Compaction chain arithmetic (verify before ANY window change)
@@ -45,14 +45,14 @@ The arithmetic (the coupled contract every endpoint's maxTokens comment
encodes):
1. **Source trigger** = `thresholdPercent × source contextWindow`
(0.70 × 200,704 = 140,492 for frank-primary on the current shape)
(0.70 × 200,704 = 140,492 for frank-archon on the current shape)
2. **Output budget** = capped at the target model's `maxTokens`
(32,768 for frank-secondary; 36,864 frank-primary; 57,344 frank-large — the
(32,768 for frank-zealot; 36,864 frank-archon; 57,344 frank-dragoon — the
largest, because halogen's `HALOGEN_MAX_TOKENS_CAP` must stay ≥ it)
3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the target's per-slot
window. Worked pairs from the live registry: 140,492 + 36,864 = 177,356 ≤
200,704 (frank-primary); 140,490 + 57,344 = 197,834 ≤ 200,700
(frank-large); 140,492 + 32,768 = 173,260 ≤ 200,704 (frank-secondary).
200,704 (frank-archon); 140,490 + 57,344 = 197,834 ≤ 200,700
(frank-dragoon); 140,492 + 32,768 = 173,260 ≤ 200,704 (frank-zealot).
Procedure (recompute from the LIVE files every time — windows rotate with
every swap; the numbers above are the 2026-10-05 state):
@@ -112,19 +112,19 @@ Use **wire names**, not OMP camelCase. Two traps:
`qwenPreserveThinking` is auto-derived in `buildOpenAICompat` from the
*auto-detected* `thinkingFormat` + local baseUrl, **before**
`applyCompatOverrides`. Setting `compat.thinkingFormat: qwen` in
models.yml does **not** enable it. Custom-named models (`frank-large`)
models.yml does **not** enable it. Custom-named models (`frank-dragoon`)
match no Qwen id pattern → use `extraBody: {preserve_thinking: true}`.
2. **Never put `chat_template_kwargs` in `extraBody`** where a compat policy
also sets it — `Object.assign` is shallow and replaces the whole object,
silently dropping the `reasoning_effort` kwarg the compat policy set.
Top-level `preserve_thinking` is honoured by halogen. (frank-primary's
Top-level `preserve_thinking` is honoured by halogen. (frank-archon's
FreeToken entry currently carries a `chat_template_kwargs` extraBody with
a comment asking to verify-and-prune — the exact trap to check at that
endpoint's next swap.)
**Gotcha: silent greedy decode.** halogen has **no server-side sampler
defaults** — `temperature` absent or `0` = greedy. If OMP sends nothing, the
endpoint decodes greedy while looking configured. This is why frank-large's
endpoint decodes greedy while looking configured. This is why frank-dragoon's
entry carries the recipe on the wire and a comment saying "do not drop it".
**Reasoning-effort folding.** Qwen3.8's template accepts only
@@ -23,7 +23,7 @@ marker. So `marker != HEAD` means "not dispatched via deploy.sh", NOT
**Why it matters:** a plain `deploy.sh` run re-selects and restarts a service
whose marker lags. On llama.cpp/R9V endpoints that is a ~16–19 min reload of
a 100+ GiB payload — and frank-primary (dankserver:7770) is also the
a 100+ GiB payload — and frank-archon (dankserver:7770) is also the
harness's own model backend, so a blind deploy is a self-inflicted outage for
possibly zero behavioral change.
@@ -62,8 +62,8 @@ service on the other host.
|---|---|---|
| Dankclaw compose services | `podman-compose -f dankclaw/compose.yaml up -d --no-deps NAME` | Recreates only `NAME`; Caddy is handled separately because it fronts both hosts. |
| SearXNG, Gluetun, Valkey, OpenWebUI, Pi-hole, Gitea database/server | user Quadlet + `systemctl --user` | Installs the selected unit and restarts only the required dependency-coupled units. Gitea's web proxy is the canonical Caddy compose service. |
| `frank-large` | systemd + `systemctl` (root unit) | Relinks and restarts only the large inference service; waits up to 20 minutes for `/health`. |
| `frank-secondary` | systemd + `systemctl` (root unit) | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. |
| `frank-dragoon` | systemd + `systemctl` (root unit) | Relinks and restarts only the large inference service; waits up to 20 minutes for `/health`. |
| `frank-zealot` | systemd + `systemctl` (root unit) | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. |
| `frank-medium` | rootless Quadlet + `systemctl --user` | Installs `dankserver/frank-medium/quadlet/frank-medium.container`; on first deploy retires the llama.cpp root unit and the vLLM replica quadlets into the home trash, then restarts the generated unit. Waits up to 20 minutes for `/health`. |
| Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver/<svc>/quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/<svc>` to `/home/dank/Containers/<svc>` (mapped-uid roots move without chown). Steady state is restart-only. |
@@ -95,7 +95,7 @@ host-side `workflows/` dir and the repo copy.
## Recommended additions
Quantized preferred, sized for the 8060S iGPU (GTT-backed; the RTX 3090 is
frank-secondary's). All repos verified 2026-10-05. None are on disk yet.
frank-zealot's). All repos verified 2026-10-05. None are on disk yet.
| # | What | Repo | Grab | Size | Why |
|---|---|---|---|---|---|
@@ -13,7 +13,7 @@ timestamp: 2026-08-26
# Dankclaw Stack
Dankclaw (192.168.0.143) runs the AI, search, development, and observability
stack. Its LLM inference endpoint `frank-large` (`:7770`) is a ROCmFPX
stack. Its LLM inference endpoint `frank-dragoon` (`:7770`) is a ROCmFPX
llama.cpp fork serving the kingjones Qwen3.8-Flash-Next-ROCmFP4-STRIX GGUF on
the card's pinned runtime; an NPU endpoint (`frank-npu`, `:7771`,
FLM v1.0.1) serves sight+sound models. The completion endpoint
@@ -30,7 +30,7 @@ graph LR
S --> V[Gluetun VPN]
O --> L[llama.cpp dankserver :7770]
O --> H[Hermes]
O --> E[frank-large :7770]
O --> E[frank-dragoon :7770]
```
## Services
@@ -48,14 +48,14 @@ graph LR
| Seerr | `seerr.franksdank.cloud` (guarded) | Caddy routes post-auth to `seerr:5055` | [[Seerr]] |
| Homepage | `franksdank.cloud` + `www.` (root site; guarded for admin actions) | served by Caddy itself from `/srv/homepage` | — |
| Pi-hole | `pihole.franksdank.cloud` (basicauth edge, not the guard chain) | admin UI `8083`; DNS on `:53` LAN-wide | [[Pi-hole]] |
| frank-large (host service) | — | `7770` | [[deployment]] |
| frank-dragoon (host service) | — | `7770` | [[deployment]] |
| frank-npu (host service) | — | `7771` (`/v1/version` health) | [[frank-npu]] |
| frank-medium (host service, Dankserver) | — | `7770` | [[frank-medium-rocmfpx]] |
The LLM rows are bare-metal systemd services, not containers. This
repo owns their scripts, units, and knowledge base (see
[LLM fleet knowledge base](../LLM/index.md)). Current mapping (2026-09-03):
`frank-large` runs the kingjones Qwen3.8-Flash-Next-ROCmFP4-STRIX GGUF
`frank-dragoon` runs the kingjones Qwen3.8-Flash-Next-ROCmFP4-STRIX GGUF
on the card's pinned `kingjones30/ROCmFPX` llama.cpp build;
`frank-npu` runs FLM v1.0.1 on the XDNA2 NPU;
`frank-medium` runs charlie12345/ROCmFPX llama.cpp serving
+3 -3
View File
@@ -47,7 +47,7 @@ is not durable. The tracked, sanitized configuration baseline is
`0.0.0.0:8080`, so the Signal HTTP API is reachable from the whole LAN.
- The data volume persists skills, memory, and session history.
- Hermes has two named local providers: `dankclaw-large` reaches
`http://192.168.0.143:7770/v1` and exposes `frank-large`;
`http://192.168.0.143:7770/v1` and exposes `frank-dragoon`;
`dankserver-small` reaches `http://192.168.0.235:7770/v1` and exposes
`frank-medium` (renamed from `frank-small` on 2026-09-15).
- The live `config.yaml` contains secrets and user state and is not committed;
@@ -64,7 +64,7 @@ is not durable. The tracked, sanitized configuration baseline is
## Gotchas
- Hermes is under active development; pin the image tag for reproducibility.
- Model choice drives quality; the local `frank-large` (Dankclaw, 7770) is the intended backend.
- Model choice drives quality; the local `frank-dragoon` (Dankclaw, 7770) is the intended backend.
- Do not expose the gateway without auth — put it behind [[Caddy]] with basic auth.
## Browser automation (Chrome DevTools Protocol daemon)
@@ -189,7 +189,7 @@ What actually breaks, and the fix:
auxiliary:
title_generation:
provider: dankserver-small # names track the live config; diff first
model: frank-secondary
model: frank-zealot
api_mode: codex_responses # real per-task key
extra_body:
reasoning:
+16 -37
View File
@@ -24,7 +24,7 @@ https://github.com/open-webui/open-webui
| Upstream latest | 0.11.4 (2026-09-21) — security/access-control advisory; `:main` is the rolling image, not a stable pin. Imagegen endpoint is `POST /api/v1/images/generations` (the `comfyui` engine is still upstream). |
| Port | 3005 (host) → 8080 (container). Port 3000 is Gitea's HTTP, so it is not usable. |
| Data volume | `/home/dank/Containers/open-webui/data` → `/app/backend/data` |
| Backends | Picker exposes two models — `frank-zealot` (host.containers.internal:7771) and `frank-dragoon` (host.containers.internal:7770, added 2026-10-09). The `frank-archon` (Dankserver) connection remains configured in the DB, disabled — see [The LLM backends](#the-llm-backends) |
| Backends | Picker exposes exactly one model — `frank-zealot` (host.containers.internal:7771). The `frank-dragoon` and `frank-archon` connections remain configured in the DB, disabled (2026-10-07) — see [The LLM backends](#the-llm-backends) |
| Network | Pinned bridge `systemd-open-webui` (`10.89.3.0/24`, gateway `10.89.3.1`) — see [History](#history) |
| Persistence | Quadlet unit `open-webui.service`; `[Install] WantedBy=default.target` |
| Deploy | `scripts/deploy-open-webui-quadlet.sh dankclaw` |
@@ -34,20 +34,17 @@ https://github.com/open-webui/open-webui
The picker is built from the `model` table in `webui.db` (one row per model)
plus the `model_ids` of each **enabled** OpenAI connection, NOT from live
connection discovery — so the picker shows exactly what is seeded there.
As of 2026-10-09 the picker exposes **two models: `frank-zealot` and
`frank-dragoon`**.
As of 2026-10-07 the picker exposes **exactly one model: `frank-zealot`**.
| Connection (`openai.api_configs` idx) | URL | Enabled | `model_ids` | Runtime | Host |
|---|---|---|---|---|---|
| 0 | `http://host.containers.internal:7770/v1` | yes (2026-10-09) | `frank-dragoon` | halogen flash-server (Strix Halo iGPU) | Dankclaw |
| 0 | `http://host.containers.internal:7770/v1` | no (2026-10-07) | `frank-dragoon` | halogen flash-server (Strix Halo iGPU) | Dankclaw |
| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | radiance TP=2 (2× R9700) | Dankserver |
| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | llama.cpp CUDA (RTX 3090, 2×200,704) | Dankclaw |
The `model` table carries one decoration row per picker model:
`frank-zealot` (context length 200,704 = the per-slot window; description:
Qwen3.5-9B Q6_K + BF16 vision mmproj) and `frank-dragoon` (context length
262,144 = the per-slot `HALOGEN_CTX`; description: Swift1.5 v2 abliterated,
MTP, vision ON).
The `model` table carries exactly one decoration row, `frank-zealot`
(context length 200,704 = the per-slot window; description: Qwen3.5-9B
Q6_K + BF16 vision mmproj).
A third gate sits *after* the table, and it is the one that bites family
accounts (2026-10-07): a model row is only visible to a non-admin user when
@@ -65,7 +62,7 @@ Two gotchas make the wiring non-obvious:
- **llama.cpp `/v1/models` is Ollama-shaped.** llama.cpp returns
`{"models":[{"name":...}]}`; OpenWebUI's `extract_data()` only reads the
OpenAI `{"data":[{"id":...}]}` shape, so connection discovery silently misses
the llama.cpp endpoints (frank-large, frank-secondary). The fix is to set
the llama.cpp endpoints (frank-dragoon, frank-zealot). The fix is to set
`model_ids` in each connection's `api_config` — when non-empty, OpenWebUI
builds the model list directly and never fetches `/v1/models`. frank-medium
(vLLM) serves the OpenAI shape and would discover fine, but uses `model_ids`
@@ -115,14 +112,11 @@ The Ollama connection is disabled (`ollama.enable=false`).
## Verification battery
1. `GET /api/models` (the picker endpoint; response key `data`) → lists
EXACTLY the intended models, **as an admin AND as a `role='user'` token**.
Do NOT gate on `GET /api/v1/models/list` (the Admin Panel → Models page):
on the 0.6.x image it returns `items: []` for connection-backed models
(workspace-list only; measured 2026-10-09 on both picker ids).
1. `GET /api/v1/models/list?limit=100` → returns EXACTLY the intended
models. Response key is `items`, not `data`.
2. `POST /api/chat/completions` per model with a trivial prompt ("Reply
with exactly one word: pong") → each returns content `pong`. Reasoning
models need a generous budget: frank-secondary (Qwen3.5 template) returns
models need a generous budget: frank-zealot (Qwen3.5 template) returns
empty content + `finish_reason: "length"` at `max_tokens 200` (the
thinking block eats it) and answers `pong` at 1500.
3. A 500 "Server Connection Error" → `podman logs open-webui` for
@@ -197,9 +191,9 @@ browser-driven and only touches the two exempt vhosts (`ai.` and `auth.`).
run with host `python3` over ssh against
`/home/dank/Containers/open-webui/data/webui.db` (no `sqlite3` CLI on
Dankclaw; table is `user`, singular; no `active` column). No per-user model
grant is needed: both picker models (`frank-zealot`, `frank-dragoon`) carry
the global `access_grant` row `(model, <id>, user, *, read)`, so any
`role='user'` account sees them. Flip on FIRST-LOGIN SIGHTING — the row does
grant is needed: the picker model `frank-zealot` carries the global
`access_grant` row `(model, frank-zealot, user, *, read)`, so any
`role='user'` account sees it. Flip on FIRST-LOGIN SIGHTING — the row does
not exist until the account signs in (2026-10-09: five provisioned accounts
had no row).
- **IdP reachability:** the container is on its own bridge (10.89.3.0/24) and
@@ -277,18 +271,18 @@ on large sizes ⇒ check ComfyUI resources; 401 ⇒ the token is bad.
- **2026-08-22 — slirp4netns → pinned bridge.** The unit ran on `Network=slirp4netns`. The slirp4netns helper process died (no OOM, no kill entry, no crash log found), and the container kept running with only `lo` in its netns: published port 3005 accepted TCP and reset every request, so chat served empty responses for ~2 days. Nothing logged an error. Moving to a kernel bridge removes the per-container helper entirely, and the healthcheck's interface check makes the zombie state visible if it ever returns.
- **2026-09-21 — model picker rebuilt for the 3-model fleet.** The picker carried 39 stale rows (37 dead Ollama models + 2 `frank-default-*`) and was missing `frank-medium`. Rebuilt the `model` table to exactly `frank-large`/`frank-medium`/`frank-secondary` and set `model_ids` on each connection (llama.cpp's Ollama-shaped `/v1/models` is invisible to OpenWebUI's discovery). Pointed the two on-host endpoints at `host.containers.internal` — the bridge refuses the host LAN IP for non-published ports. All three verified end-to-end via `/openai/chat/completions`.
- **2026-09-21 — model picker rebuilt for the 3-model fleet.** The picker carried 39 stale rows (37 dead Ollama models + 2 `frank-default-*`) and was missing `frank-medium`. Rebuilt the `model` table to exactly `frank-dragoon`/`frank-medium`/`frank-zealot` and set `model_ids` on each connection (llama.cpp's Ollama-shaped `/v1/models` is invisible to OpenWebUI's discovery). Pointed the two on-host endpoints at `host.containers.internal` — the bridge refuses the host LAN IP for non-published ports. All three verified end-to-end via `/openai/chat/completions`.
- **2026-09-24 — SSO via authentik.** OIDC authorization-code flow wired per the [SSO (authentik)](#sso-authentik) section: application/provider/entitlements created in authentik, client secret in the host env file, roles from the `roles` scope claim. E2E coverage in `tests/e2e/specs/18-openwebui-sso.spec.ts`.
- **2026-09-24 — flipped to SSO-only.** The OIDC round-trip verified in a real browser (login view renders no password form, only "Continue with authentik", and the flow lands on a signed-in session), so the local username/password path was retired: `ENABLE_LOGIN_FORM=false` + `ENABLE_PASSWORD_AUTH=false` in the quadlet and `ui.enable_login_form` set to `false` in the config DB (the DB wins over env, so both were changed together). `qa-e2e-admin` was demoted out of the `authentik Admins` group (it keeps `openwebui_users` / `homepage-admins` for app access).
- **2026-10-07 — picker narrowed to `frank-secondary` only.** Per Frank: the only model accessible in OpenWebUI is `frank-secondary`. Rebuilt the `model` table to a single `frank-secondary` row (the old rows `frank-large`/`frank-primary`/`frank-small` are deleted), set `openai.api_configs` indices 0 (frank-large) and 1 (frank-primary) to `enable: false`, and retargeted index 2's `model_ids` from the stale `frank-small` label to `frank-secondary` — llama.cpp serves by alias, so the old label addressed a model name the server does not know. Pre-change snapshot: `/home/dank/Containers/open-webui/data/webui.db.bak-20261007` (+ `-wal`/`-shm`). Verified: `/api/v1/models/list` returns exactly `["frank-secondary"]`, chat completion answers `pong` (needs `max_tokens≥1500` — the Qwen3.5 thinking block consumes a 200-token budget), and anonymous API probes are all 401 (see [API access](#api-access-auth-gated)).
- **2026-10-07 — picker narrowed to `frank-zealot` only.** Per Frank: the only model accessible in OpenWebUI is `frank-zealot`. Rebuilt the `model` table to a single `frank-zealot` row (the old rows `frank-dragoon`/`frank-archon`/`frank-small` are deleted), set `openai.api_configs` indices 0 (frank-dragoon) and 1 (frank-archon) to `enable: false`, and retargeted index 2's `model_ids` from the stale `frank-small` label to `frank-zealot` — llama.cpp serves by alias, so the old label addressed a model name the server does not know. Pre-change snapshot: `/home/dank/Containers/open-webui/data/webui.db.bak-20261007` (+ `-wal`/`-shm`). Verified: `/api/v1/models/list` returns exactly `["frank-zealot"]`, chat completion answers `pong` (needs `max_tokens≥1500` — the Qwen3.5 thinking block consumes a 200-token budget), and anonymous API probes are all 401 (see [API access](#api-access-auth-gated)).
- **2026-10-09 — re-keyed to the Protoss ids (wire rename).** The endpoints
began serving `frank-zealot` / `frank-dragoon` / `frank-archon`, so three
id-keyed objects had to move in the same window as the alias flip:
`model.id`+`name` (`frank-secondary` → `frank-zealot`, `params`
`model.id`+`name` (`frank-zealot` → `frank-zealot`, `params`
`{"context":{"length":200704}}` and the `meta.description` kept),
`access_grant` (`resource_type='model'`, `resource_id`, principal
`user`/`*`, permission `read`), and `openai.api_configs` per-backend
@@ -314,21 +308,6 @@ on large sizes ⇒ check ComfyUI resources; 401 ⇒ the token is bad.
name: /selected model/i })`) and the welcome text "How can I help you
today?"; that snapshot is also the cheapest proof the picker is populated.
- **2026-10-09 — `frank-dragoon` added to the picker.** Per Frank: expose the
halogen endpoint (service `frank-large`, `host.containers.internal:7770`)
beside `frank-zealot`. Three id-keyed writes (the trio the 10-07 rebuild
proved): `openai.api_configs` idx 0 `enable: true` (its `model_ids` already
carried `frank-dragoon` from the wire rename), a `model` row (`id`/`name`
`frank-dragoon`, `params.context.length` 262144 = the per-slot
`HALOGEN_CTX`, description of the v2 abliterated payload), and the global
`access_grant` `(model, frank-dragoon, user, *, read)`. Snapshot
`webui.db.bak-20261009b`, apply with host `python3` over ssh, then
`systemctl --user restart open-webui.service`. Verified: `/api/models`
lists `frank-dragoon` as admin AND as the QA `role='user'` account, and
`POST /api/chat/completions` through the app answers `pong` at
`max_tokens 1500`. Halogen's `/v1/models` is OpenAI-shaped, so discovery
would have worked too — `model_ids` stays authoritative.
## See Also
- [[Dankclaw Stack]]
@@ -33,10 +33,10 @@ it is not invisible.
## Why 64 GiB
Dankclaw is shared. `frank-large` alone holds **103.5 GiB of the 121.2 GiB**
Dankclaw is shared. `frank-dragoon` alone holds **103.5 GiB of the 121.2 GiB**
host (`68.0 weights + 14.4 KV pool + 21.1 working`), leaving roughly
**7.1 GiB** for 28 co-tenant containers whose working set is ~4.3 GiB. The
engine itself warns that when that runs out, frank-large and its neighbours
engine itself warns that when that runs out, frank-dragoon and its neighbours
"can stall for minutes at a time at full CPU with no disk activity and no
output". Swap is the valve that turns an OOM kill into slow paging.
@@ -96,20 +96,20 @@ Then confirm the inference service is unaffected — a completion, not just
```bash
curl -s http://127.0.0.1:7770/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"frank-large","messages":[{"role":"user","content":"Reply with exactly: SWAP64-OK"}],"max_tokens":48,"reasoning_effort":"low"}'
-d '{"model":"frank-dragoon","messages":[{"role":"user","content":"Reply with exactly: SWAP64-OK"}],"max_tokens":48,"reasoning_effort":"low"}'
```
As run 2026-09-12: `finish_reason: stop`, `SWAP64-OK`, 28 containers up,
frank-large `health=200` throughout.
frank-dragoon `health=200` throughout.
## Caveats
- **Swap cannot help the pinned weights.** frank-large's ~68 GiB of weights are
- **Swap cannot help the pinned weights.** frank-dragoon's ~68 GiB of weights are
held deliberately; paging them is a stall, not a relief. More swap does not
make the doubled KV pool affordable — see the reserve/pool ceiling in
[[halogen-flash-server]].
- **`vm.swappiness=60` is untested against this workload.** Under real
pressure the kernel will happily page anonymous memory, and frank-large's KV
pressure the kernel will happily page anonymous memory, and frank-dragoon's KV
pool and working set *are* anonymous. Swapping the KV pool would be far worse
than the OOM kill it prevents. If the box ever starts thrashing, lowering
swappiness (or capping the engine's cgroup) is the knob, not more swap.
@@ -22,8 +22,8 @@ The homepage is live on Dankclaw. `./scripts/deploy.sh dankclaw` copies the stat
**Docs page shipped (2026-09-17):** `/docs` serves a static user documentation
page (`shared/homepage/docs.html`) — access walkthrough, the running-model
guide, media how-tos, and troubleshooting. Per-service `/docs/<svc>/` pages
remain a separate thread. The three LLM cards (`frank-large`, `frank-medium`,
`frank-secondary`) moved from admin-only to the **user** page with live model
remain a separate thread. The three LLM cards (`frank-dragoon`, `frank-medium`,
`frank-zealot`) moved from admin-only to the **user** page with live model
descriptions and a `docs` link to `/docs#models`.
**Sections + WIP split (2026-09-21):** both pages now group their panels
@@ -48,23 +48,23 @@ exact bug bit the screenshots on 2026-10-08). The docs page gained a full
(`/docs#api-access`): per-user OpenWebUI API keys (User menu → Settings →
Account → API keys), the base URL `https://ai.franksdank.cloud/api/v1`, and
copy-paste configs for **Opencode** (`~/.config/opencode/opencode.json`) and
**Pi** (`~/.pi/agent/models.json`) pointing at `frank-secondary`. Shipped
**Pi** (`~/.pi/agent/models.json`) pointing at `frank-zealot`. Shipped
together with the OpenWebUI side: `features.api_keys=true` for regular users
and the restored `frank-secondary` model row + public `read` access grant
and the restored `frank-zealot` model row + public `read` access grant
without which the picker showed "No results found" for every family account
(see [[OpenWebUI]]).
**Capability equivalence notes (2026-10-08):** each fleet card (and the
`/docs#models` section) carries a `details.equiv` line — frank-large
`/docs#models` section) carries a `details.equiv` line — frank-dragoon
"equivalent to Claude Sonnet 5 / OpenAI-Luna (Summer 2026 frontier
intelligence)", frank-primary "equivalent to Claude Opus 4.8 / OpenAI-Terra
(same era)", frank-secondary "equivalent to last year's frontier level
intelligence)", frank-archon "equivalent to Claude Opus 4.8 / OpenAI-Terra
(same era)", frank-zealot "equivalent to last year's frontier level
(2025)". Rendered by `descHTML()` as `.desc-equiv` (head color, italic);
pinned in `homepage_config.bats`.
**Protoss rename + Model Information component (2026-10-08):** the model
cards became the StarCraft trio — `frank-large` → **frank-dragoon**
(iGPU), `frank-primary` → **frank-archon** (Dankserver), `frank-secondary`
cards became the StarCraft trio — `frank-dragoon` → **frank-dragoon**
(iGPU), `frank-archon` → **frank-archon** (Dankserver), `frank-zealot`
→ **frank-zealot** (3090). The endpoint/API ids did NOT change in this wave; only the
family-facing names. The three cards fused into one **Model Information**
component (`app.js` `renderModelInfo()`, `.model-info`/`.mi-*` styles):
@@ -201,8 +201,8 @@ the pixels.
**Model ids official (2026-10-08):** the card ids `frank-zealot`,
`frank-dragoon`, `frank-archon` are now the harness's official model ids
in `models.yml`/`config.yml` (see [[OMP Harness]]); the
systemd services keep the `frank-primary`/`frank-large`/
`frank-secondary` names. Homepage, docs page and harness now agree on
systemd services keep the `frank-archon`/`frank-dragoon`/
`frank-zealot` names. Homepage, docs page and harness now agree on
one name per model.
**Api ids renamed on the wire (2026-10-09):** the promise in
@@ -316,7 +316,7 @@ flowchart TD
## Service classification
**User services** (root page), in section order:
- **AI** — OpenWebUI, and the three LLM fleet cards (frank-large, frank-primary, frank-secondary — status + docs link, no Open link).
- **AI** — OpenWebUI, and the three LLM fleet cards (frank-dragoon, frank-archon, frank-zealot — status + docs link, no Open link).
- **Web Apps** — SearXNG, Gitea, Copyparty, Marreta.
- **Media Library Apps** — Calibre-Web, RomM, Plex, Jellyfin, Seerr.
@@ -38,7 +38,7 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
| `node-dankserver` | `192.168.0.235:9100` | host metrics from [[Node-Exporter]] — a different machine, so the LAN address works |
| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network; both serve real `/metrics` |
| `frank-medium` | `192.168.0.235:7770` | the LLM fleet — `vllm:` series; a different machine, so the LAN address works |
| `frank-large`, `frank-secondary` | `host.containers.internal:7770`, `:7771` | the LLM fleet — `llamacpp:` series; same-host net=host binds, so via the gateway |
| `frank-dragoon`, `frank-zealot` | `host.containers.internal:7770`, `:7771` | the LLM fleet — `llamacpp:` series; same-host net=host binds, so via the gateway |
**Three jobs read down, and they are not outages** (audited 2026-09-16):
@@ -115,7 +115,7 @@ unconditionally, while **context-backed** types (`draft-mtp`, `draft-dflash`,
- Internal order is fixed (ngram types first, then drafters); CLI order is
irrelevant.
- **A legal chain can still lose on a recurrent model.** `draft-mtp,ngram-mod`
on the qwen35 hybrid (frank-large era) measured −15 % prose / −7 %
on the qwen35 hybrid (frank-dragoon era) measured −15 % prose / −7 %
block-copy (2026-10-03): adding the ngram layer logs `recurrent state
rollback is not compatible with 'ngram-mod' - disabling rollback support`,
and MTP verification depends on that rollback, so acceptance drops on ALL
@@ -215,9 +215,9 @@ The DFlash draft is not a standalone model. It must pair with its target model.
## Deployment on Dankserver (NVIDIA RTX 3090)
The Dankserver llama.cpp deployment (`frank-secondary`, Qwen3.6-35B-A3B) is **active** on Dankserver (port 7770) as of the 2026-08-17 redo (the earlier 2026-08-16/17 retirement of this service was reversed when the PP split was retired). The vLLM material below documents the archived AWQ/DFlash path on the same hardware.
The Dankserver llama.cpp deployment (`frank-zealot`, Qwen3.6-35B-A3B) is **active** on Dankserver (port 7770) as of the 2026-08-17 redo (the earlier 2026-08-16/17 retirement of this service was reversed when the PP split was retired). The vLLM material below documents the archived AWQ/DFlash path on the same hardware.
> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-secondary` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only.
> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-zealot` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only.
The vLLM material below documents the archived AWQ/DFlash path on the same hardware. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions:
@@ -250,7 +250,7 @@ docker run --rm --name vllm-qwen36-dflash \
## Deployment on Dankclaw (AMD Strix Halo gfx1151)
`frank-large` on Dankclaw ran the llama.cpp pipeline-parallel head (Qwen3.8-27B-Uncensored Q6_K) on port 7770, RPC-split with the Dankserver worker. Spec was the standalone MTP draft over the ngram-mod fallback, 1 slot at 262144 ctx, q8_0/f16 KV, head GPU side ROCm0. This page is a historical record of that era (mid-August 2026); for the current deployment see [[kingjones-strix-rocmfp4]].
`frank-dragoon` on Dankclaw ran the llama.cpp pipeline-parallel head (Qwen3.8-27B-Uncensored Q6_K) on port 7770, RPC-split with the Dankserver worker. Spec was the standalone MTP draft over the ngram-mod fallback, 1 slot at 262144 ctx, q8_0/f16 KV, head GPU side ROCm0. This page is a historical record of that era (mid-August 2026); for the current deployment see [[kingjones-strix-rocmfp4]].
History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 2026-08-15 after a ROCm illegal-memory-access crash under two concurrent 16k prompts) and before that bundled MTP (`draft-mtp`, retired same day). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`), the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`), and the DSpark draft (`Qwen3.8-27B-DSpark-Q8_0.gguf`) remain in `/home/dank/models/` as reference only.
@@ -315,7 +315,7 @@ not switch without re-measuring".
The A/B vehicle for ANY single-GPU llama.cpp knob (batch/ubatch, KV type,
slot split, quant rung — not just spec flags) is the transient systemd
drop-in procedure in [[frank-large-benchmark]] and [[frank-secondary-benchmark]].
drop-in procedure in [[frank-dragoon-benchmark]] and [[frank-zealot-benchmark]].
## Is it worth it? (arithmetic before deployment)
@@ -419,4 +419,4 @@ measurement that would settle it.
- [[GGUF Compatibility]] — header parsing and custom tensor-type lock-in
- [[Llama Cpp Serving Sizing]] — slot-pool and drafter KV sizing arithmetic
- [[Llama Cpp Multi-Device Splits]] — where the drafter's KV actually lands
- [[frank-large-benchmark]] and [[frank-secondary-benchmark]] — the single-GPU env drop-in A/B vehicle
- [[frank-dragoon-benchmark]] and [[frank-zealot-benchmark]] — the single-GPU env drop-in A/B vehicle
@@ -22,7 +22,7 @@ marked "verify first". Re-verified 2026-08-15/16; corrections marked below.
(b10433): Vulkan 25.13 t/s vs ROCm 24.33 t/s decode at 16k; at 262k ctx
Vulkan decodes 48% faster (5.68 vs 3.83 t/s). ROCm wins prefill (410 vs
302 t/s @64k). The "HIP out-performs Vulkan for decode" claim did not
hold on the 8060S. See [[frank-large-benchmark]].
hold on the 8060S. See [[frank-dragoon-benchmark]].
- **Power profile**: the `/sys/class/drm/card0/device/power_profile` file
does not exist on this machine (checked 2026-08-15). GPU DPM is already
`performance`, forced level `auto`. This item is N/A.
@@ -48,7 +48,7 @@ lifted to 114 GiB. ROCm/SMI still reports a 96 GiB VRAM carve-out (VGM) — a
different, driver-reported figure that has always been larger than the kernel
cap.
This split is the Strix Halo GTT/VGM design. It drives the batch/ubatch memory
limits documented in [[frank-large-benchmark]].
limits documented in [[frank-dragoon-benchmark]].
## 1. Self-Hosted Agentic Tools
@@ -106,7 +106,7 @@ All require root and a reboot for the modprobe change.
## Sources
- [[frank-large-benchmark]] — measured backend/spec numbers
- [[frank-dragoon-benchmark]] — measured backend/spec numbers
- [[deployment]] — deploy and update process
- AMD ROCm RDNA3.5 system optimization guide (gfx1151 memory/BIOS)
- llama.cpp source: `ggml/CMakeLists.txt`, `ggml/src/ggml-hip/CMakeLists.txt`,
@@ -115,7 +115,7 @@ All require root and a reboot for the modprobe change.
## See Also
- [[frank-large-benchmark]]
- [[frank-dragoon-benchmark]]
- [[DFlash DSpark Speculative Decoding]]
- [[deployment]]
- [[index]]
@@ -46,7 +46,7 @@ Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. The host run
| Runtime | FastFlowLM 0.9.45 + XRT (installed 2026-08-16; **retired 2026-09-11** with the `amd_iommu=off` decision) |
| Status | Not in service. Research only. See [[FastFlowLM NPU Inference]] |
The NPU is retired (2026-09-11). The `frank-secondary` name now serves Gemma4-12B QAT on the RTX 3090 via llama.cpp (see [[frank-secondary-gemma4-12b]]).
The NPU is retired (2026-09-11). The `frank-zealot` name now serves Gemma4-12B QAT on the RTX 3090 via llama.cpp (see [[frank-zealot-gemma4-12b]]).
### ROCm
@@ -101,7 +101,7 @@ cable between claw and server is required for 10G.
- Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
- The GPU budget is the GTT aperture: ~114 GiB (`gtt_total` in `/sys/class/drm/card1/device/`; `llama-server --list-devices` reports 116,732 MiB). rocm-smi's smaller SVM/dedicated carve-out is **not** the budget — the 2026-08 "64 GiB rocm-smi" lesson is stale. Do not use `free` or `dmidecode` for the GPU budget.
- The active service is `frank-large` on port 7770: **gufo** (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source at `/home/dank/GitPackages/gufo-v0.1.1/`, serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, DFlash2 drafter). See [[frank-large-gufo-qwen38-27b]].
- The active service is `frank-dragoon` on port 7770: **gufo** (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source at `/home/dank/GitPackages/gufo-v0.1.1/`, serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, DFlash2 drafter). See [[frank-dragoon-gufo-qwen38-27b]].
- The active runtime **links against the host ROCm** (`/opt/rocm` → `/etc/alternatives/rocm` → 7.2.4) through its RPATH, so it needs the host HIP stack rather than a bundled one — the opposite of the retired llama.cpp prebuilt, which carried its own. Building it needs the ROCm dev packages that are installed, and two workarounds recorded in the runbook (drop `librocwmma-dev`; pin `--gcc-install-dir` to GCC 15). The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) is not in the serving path.
- The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
- 1.1 TB free disk space is ample for model storage.
@@ -17,7 +17,7 @@ timestamp: 2026-09-20
Dankserver is a media/library server and the `frank-medium` LLM host. The
host runs Ubuntu 26.04 (Resolute Raccoon). The RTX 3090 that served the
FreeToken/CUDA `frank-secondary` era was removed on 2026-09-15 and replaced with
FreeToken/CUDA `frank-zealot` era was removed on 2026-09-15 and replaced with
two AMD Radeon AI PRO R9700 cards.
## Hardware Specifications
@@ -107,8 +107,8 @@ hijack `libhsa-runtime64` were purged on 2026-09-15 — see
The active inference service is `frank-medium.service`: mainline llama-server on
port 7770, layer-split 32:33 across both R9700s with the DFlash2 drafter pinned to
ROCm1. See [[frank-medium-llamacpp-dflash2]]. The former
`frank-secondary.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU swap;
its unit is archived in `~/.AGENTTRASH/20260915-frank-secondary-retire/host-units/`,
`frank-zealot.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU swap;
its unit is archived in `~/.AGENTTRASH/20260915-frank-zealot-retire/host-units/`,
and the vllm-radiance replica trio in `~/.AGENTTRASH/20260916-vllm-radiance-retire/`.
See [[deployment]].
@@ -13,12 +13,12 @@ timestamp: 2026-09-03
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
> See [[frank-redo-2026-08-17]] and [[deployment]] for the active configuration.
> The `frank-secondary` name and port 52625 then belonged to the vLLM
> The `frank-zealot` name and port 52625 then belonged to the vLLM
> service on Dankclaw (nameistoken Quark W8A8); this page is archive.
# FastFlowLM NPU Inference
FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and `frank-secondary` (port 52625) serves in production. See Promotion.
FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and `frank-zealot` (port 52625) serves in production. See Promotion.
## Verdict (2026-08-16)
@@ -167,7 +167,7 @@ Proposed role: a small, always-on, low-power endpoint for triage, summarization,
- `gpt-oss:20b` pulled (4/4 files verified, 13.8 GB `model.q4nx` + tokenizer) into `~/.config/flm/models/GPT-OSS-20B-NPU2/`.
- `qwythos:9b` (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2) installed 2026-08-17 via `flm-add.py` (downloaded from the HF repo; stdlib-only, no pip needed). Tag registered in the user registry `~/.config/flm/model_list.json` (a copy of the v1.0.1 system registry + the custom entry); kernels symlinked `~/.config/flm/xclbins/Huihui-...-NPU2 -> /opt/fastflowlm/share/flm/xclbins/Qwen3.5-9B-NPU2`. The service env exports `FLM_CONFIG_PATH` and `FLM_XCLBIN_PATH` so the tag resolves (without them flm hits the unknown-tag fallback, see the incident below).
- `flm serve gpt-oss:20b --host 0.0.0.0 -p 52625 --pmode performance` runs as a nohup process. It is not a systemd unit. It does not survive a reboot.
- The `llama-server` (frank-large, the PP head) holds an fd on `/dev/accel/accel0` from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16.
- The `llama-server` (frank-dragoon, the PP head) holds an fd on `/dev/accel/accel0` from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16.
## Measured Results (2026-08-16, gpt-oss-20b on the Dankclaw NPU)
@@ -185,7 +185,7 @@ Long-context prefill is not yet measured (runs 1-3 used under 200 prompt tokens)
## Model Bakeoff (measured 2026-08-16, 4 models)
All runs on the Dankclaw NPU (50 TOPS Strix Halo), single stream, `--pmode performance`, PP head (frank-large) serving concurrently on 7770. Decode t/s is the FLM-reported rate. Vision accuracy = ground-truth elements out of 4 (text SPECTRUM, red circle left, blue square right, three green triangles bottom).
All runs on the Dankclaw NPU (50 TOPS Strix Halo), single stream, `--pmode performance`, PP head (frank-dragoon) serving concurrently on 7770. Decode t/s is the FLM-reported rate. Vision accuracy = ground-truth elements out of 4 (text SPECTRUM, red circle left, blue square right, three green triangles bottom).
| Model | Tag | Weights | Decode t/s (short) | 8k prefill t/s | 8k decode t/s | Vision decode t/s | Vision accuracy | Max ctx (registry) |
|-------|-----|---------|-------------------:|---------------:|--------------:|------------------:|----------------|--------------------|
@@ -203,21 +203,21 @@ Notes:
## Promotion (2026-08-16)
`frank-secondary` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-secondary.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-secondary.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch.
`frank-zealot` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-zealot.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-zealot.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch.
### Boot policy (2026-08-17)
The `frank-secondary` unit is disabled for boot. Only the PP pair (`frank-large` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:
The `frank-zealot` unit is disabled for boot. Only the PP pair (`frank-dragoon` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:
```
sudo ln -sf /home/dank/vllmScripts/scripts/frank-secondary.service /etc/systemd/system/frank-secondary.service
sudo ln -sf /home/dank/vllmScripts/scripts/frank-zealot.service /etc/systemd/system/frank-zealot.service
sudo systemctl daemon-reload
sudo systemctl enable frank-secondary
sudo systemctl enable frank-zealot
```
Live config (2026-08-16): `FLM_PMODE=turbo`, `FLM_ASR=1` (loads Whisper-V3-Turbo-NPU2; exposes `/v1/audio/transcriptions`), `FLM_CTX_LEN=131072` (verified max — Gemma4 natively supports 128K and the NPU budget fits it; KV occupancy measured at 0.11% for a 123-token exchange, decode 12.0 t/s, 13 GiB headroom).
The PP split was renamed `frank-pp-head` → `frank-large` (2026-08-16, commit `48142f5`).
The PP split was renamed `frank-pp-head` → `frank-dragoon` (2026-08-16, commit `48142f5`).
Model swap (2026-08-16, commit `50c55ee`): `FLM_MODEL=qwen3.6-moe:35b-a3b`, `FLM_CTX_LEN=262144` (native max), `FLM_PMODE=turbo`, `FLM_ASR=1` kept. Decode measured 12.5-13.1 t/s. The Gemma4 live config above (131072 ctx) applied to `gemma4-it:e4b`. Swap `FLM_MODEL` and `FLM_CTX_LEN` back to restore it.
@@ -353,7 +353,7 @@ installed) against `/opt/fastflowlm/share/flm/xclbins/` (available).
Symptom: a client on dankware could not reach `qwen3.6-moe:35b-a3b` on port 52625. The journal shows three separate causes in the 14:19-15:03 window.
1. **EXEC 203 crash loop (14:19-14:20).** The exec bit was lost on `serve-flm-small.sh` on the target. The unit failed `status=203/EXEC: Permission denied` four times, then started after the bit was restored. The git mode is `100755` (commit `3f42210`), and a `git pull` restores the bit. Verify with `git ls-files -s scripts/serve-flm-small.sh`.
2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-secondary` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-secondary' is not supported.` Each such request also triggered the unknown-tag fallback below.
2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-zealot` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-zealot' is not supported.` Each such request also triggered the unknown-tag fallback below.
3. **Swap churn (14:48-15:03).** The swap `gemma4-it:e4b` → `qwen3.6-moe:35b-a3b` restarted the service several times. Each load of the 23 GB model takes about 1.5-2.5 minutes at 262144 ctx. The endpoint does not serve during a load.
Fix: the client model ID must be the FLM registry tag `qwen3.6-moe:35b-a3b`. The working pi agent entry in `~/.pi/agent/models.json` is `baseUrl http://192.168.0.143:52625/v1`, `id qwen3.6-moe:35b-a3b`, `contextWindow 262144`, `reasoning true`, `input [text, image]`. Verified end-to-end from dankware (192.168.0.226) at 15:32 on 2026-08-16.
@@ -390,7 +390,7 @@ pi sessions failed against this endpoint with `Stream ended without finish_reaso
- **The FLM serve process is not under systemd.** A reboot of Dankclaw kills it. The PP head service is enabled and returns. The FLM process must be restarted by hand. A unit file is a natural next step.
- **gpt-oss reasoning budget.** The model can spend its full token budget on `reasoning_content` and return empty content. Clients must handle or cap this.
- **No llama.cpp GGUF path.** FLM uses its own pre-compiled kernels. Our UD-Q8_K_XL and HauhauCS builds are not importable. NPU work means FLM-specific model tags.
- **NPU production role (2026-08-16).** `frank-secondary` serves `qwen3.6-moe:35b-a3b` on port 52625 (swap churn and the unknown-tag incident are recorded above). The remaining NPU content (benchmarks, runbook steps) is still research until a step lands and is recorded.
- **NPU production role (2026-08-16).** `frank-zealot` serves `qwen3.6-moe:35b-a3b` on port 52625 (swap churn and the unknown-tag incident are recorded above). The remaining NPU content (benchmarks, runbook steps) is still research until a step lands and is recorded.
## Citations
@@ -23,7 +23,7 @@ read the returned content:
```python
import json, urllib.request
req = urllib.request.Request("http://<host>:<port>/v1/chat/completions",
json.dumps({"model": "frank-secondary", "messages": [{"role": "user",
json.dumps({"model": "frank-zealot", "messages": [{"role": "user",
"content": "Hi"}]}).encode(), {"Content-Type": "application/json"})
print(json.load(urllib.request.urlopen(req, timeout=600)))
```
@@ -119,5 +119,5 @@ re-download.
- [[Llama Cpp Runtimes]] — which fork ships which custom types
- [[Llama Cpp Code Diff]] — the enum tables and code-level differences
- [[frank-large-rocmfpx-q8agent-mtp]] — the ftype-115 lock-in in production
- [[frank-dragoon-rocmfpx-q8agent-mtp]] — the ftype-115 lock-in in production
- [[Llama Cpp Serving Sizing]] — pricing what a payload costs once it loads
@@ -13,7 +13,7 @@ Fleet law first: **STOP the production service before any GPU experiment**
GPU and corrupts results; an unisolated GPU experiment has precedent for
wedging a host until power cycle. Reference harnesses:
`scripts/bench-batch-ubatch.sh`, `scripts/autoresearch.sh`,
`scripts/autoresearch/frank-primary-arms.sh`.
`scripts/autoresearch/frank-archon-arms.sh`.
## 1. The window structure (one config per arm, a driver owns the window)
@@ -27,7 +27,7 @@ wedging a host until power cycle. Reference harnesses:
with `container state improper`.
2. **Boot the arm as the service, not a scratch port** (no scratch ports on
LLM hosts — one card, one server). The serve script takes its config via
env-file override (`FRANK_SECONDARY_ENV_FILE`, `FRANK_PRIMARY_AR_ENV_FILE`)
env-file override (`FRANK_ZEALOT_ENV_FILE`, `FRANK_ARCHON_AR_ENV_FILE`)
pointing at a staged copy; the checkout is never mutated.
3. **Fast-fail the health wait**: poll `podman ps | grep -qx <name>` inside
the loop — a refused boot dies in one poll instead of 20 min. Capture the
@@ -245,17 +245,17 @@ nearly shipped the wrong setting.
the comment); verify multi-line command construction with `bash -n` AND a
render test.
## 8. Results location and recovery (frank-primary arms)
## 8. Results location and recovery (frank-archon arms)
| copy | path |
|---|---|
| per-arm evidence | `$ARMS_DIR` = `~/.jobs/<run-stamp>/` (`.rc`, `.done` beside it) |
| summary | `$ARMS_DIR/summary.tsv` = `arm rc decode_tps decode_tps_t0.7 kv_tokens boot_s` + `<arm>.out/.env/.overlay/.rc/.metrics` |
| server-side last window | `/home/dank/.autoresearch-frank-primary/run.json` |
| server-side last window | `/home/dank/.autoresearch-frank-archon/run.json` |
| durable memo | `.../last/run.json` + `last/hash` (promoted only after bench AND verified restore) |
| fallback | `~/.omp/agent/sessions/<ws>/<session>/<N>.{shake,bash}.log` (permanent) |
`/tmp/frank-primary-arms-*` and `/tmp/armstub/` are stub self-tests —
`/tmp/frank-archon-arms-*` and `/tmp/armstub/` are stub self-tests —
constants 99.9/88.8 identical across arms by construction; **identical
numbers across arms == stub or bug, never a measurement.** After compaction,
re-harvest with `grep -rhoE 'decode_tps[a-z0-9_.]*=[0-9.]+'` over the session
@@ -161,8 +161,8 @@ of the same RAM, not extra memory.
## 7. IBS model-access attribution (AMD CPUs)
Real CPU access patterns for mmap'd weights, live, no stop (established on
frank-primary, 2026-09-26, ~2.7M attributed samples in 75 s). Reference:
`scripts/frank-primary-expert-activity.py` (self-contained dump + sample +
frank-archon, 2026-09-26, ~2.7M attributed samples in 75 s). Reference:
`scripts/frank-archon-expert-activity.py` (self-contained dump + sample +
analyze; `--analyze-only`).
1. GGUF header-only scan (ms for a 50 GiB part) → tensor extents. **Extents
@@ -80,7 +80,7 @@ co-tenants.
- **Traps:** mmap'd PLE is reclaimable page cache; `-ot` makes it pinned —
"move layers to the card" converts reclaimable cache into device memory and
frees nothing for co-tenants. A served model's footprint ≠ weight size
(frank-secondary: 22.6 of 24.1 GiB for an 11.29 GiB model). Do the capacity
(frank-zealot: 22.6 of 24.1 GiB for an 11.29 GiB model). Do the capacity
subtraction before designing: 12 attn layers × 1.62 GiB + 200k f16 KV =
28.6 GiB > 24.1 GiB usable → KV-in-VRAM on that pair only reaches ~64k.
- **Context lengths: derive, never remember.** `llama-bench` computes
@@ -157,7 +157,7 @@ in the timings. Context cap for full-model residency on that shape stayed
## Related
- [[GGUF Compatibility]] — what a payload is before it is priced
- [[frank-secondary-llama-mimo-q6k-2x200704]] — the current two-boot priced shape
- [[frank-zealot-llama-mimo-q6k-2x200704]] — the current two-boot priced shape
- [[frank-medium-llamacpp-dflash2]] — where the 68 KiB/token datum was measured
- [[DFlash DSpark Speculative Decoding]] — the spec A/B protocol on llama.cpp
- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — the multi-device side
@@ -3,15 +3,24 @@ type: reference
title: Model Swap and Rollback
description: The full model-swap discipline for the LLM fleet — fit preflight, fetch and payload verification, runtime build/flag checks, git-only config, live validation, systemd handoff, close-out, the three-pin revert law, and research-verification discipline.
tags: [models, swap, gguf, huggingface, rollback, deployment, verification]
timestamp: 2026-10-05T00:00:00-07:00
timestamp: 2026-10-15T00:00:00-07:00
---
# Model Swap and Rollback
Applies to every endpoint swap on the fleet (frank-secondary / frank-large /
frank-primary). Repo law: all config flows through git; the live models are
Applies to every endpoint swap on the fleet (frank-zealot / frank-dragoon /
frank-archon). Repo law: all config flows through git; the live models are
host state under `/home/dank/models/<name>/`.
**No bench step (law, 2026-10-15).** A model swap NEVER schedules a bench
window. Performance evidence comes from the deployed service itself: the
engine's boot ledger, per-request timings (`/completion` `timings`,
`Server-Timing`), and one real request per harness role (§6, §8). A
deliberate standalone bench (flag A/B, shape sweep) is OUT OF PROCEDURE —
a separate operation that follows the AGENTS.md Benchmarking Policy
(stop & mask). The env-file hook (§5) stays so those out-of-procedure
arms remain possible.
## 1. Fit preflight — never deploy a swap without confirming it FITS
1. Get the checkpoint's on-disk/weight size AND precision from the HF card
@@ -144,7 +153,8 @@ recoverable, drift is not.
## 8. Close-out (mandatory)
1. Docs: active runbook updated (build recipe, profile, validation numbers
incl. real-workload prefill/decode); superseded runbook gets a RETIRED
incl. real-workload prefill/decode measured on the LIVE service — no
bench window); superseded runbook gets a RETIRED
banner and stays as the rollback record; `LLM/index.md` mermaid + active
list; AGENTS.md fleet table + runtime section + history chain; work-log
entry in `serverAppConfigs-docs/log.md`.
@@ -277,5 +287,6 @@ predecessor "stopped" (the port race above).
- [[GGUF Compatibility]] — which runtime can load a payload at all
- [[Llama Cpp Serving Sizing]] — pricing the shape once it loads
- [[RDNA4 MoE Expert Offload VRAM Budget]] — vLLM-side fit arithmetic
- [[LLM Bench Windows and Measurement]] — measuring the new deployment
- [[LLM Bench Windows and Measurement]] — out-of-procedure bench windows
only; a new deployment is measured by §6 live validation, not by a bench
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the current endpoint
@@ -26,7 +26,7 @@ conclusions.
| Blocks | 65 (45 on worker, 20 on head) | `--tensor-split 0.68,0.32` |
| FA layers | 16 total, interval 4 (11 worker / 5 head) | GGUF `full_attention_interval` |
| KV per FA layer per token | 3136 B (K q8_0 1088 + V f16 2048) | env file comment, confirmed |
| ubatch | 256 | `UBATCH_SIZE` in `frank-large.env` |
| ubatch | 256 | `UBATCH_SIZE` in `frank-dragoon.env` |
| MTP verify step positions | 5 (1 + n_max 4) | `SPEC_DRAFT_N_MAX=4` |
| Link (old path) | 1 GbE via USB-C adapter on claw | `enx00e04cc2e1e5`, negotiated 1000 Mb/s |
| Link (current path) | 2.5 GbE eno1 to eno1 | negotiated 2500 Mb/s both ends |

Some files were not shown because too many files have changed in this diff Show More