diff --git a/scripts/frank-archon.env b/scripts/frank-archon.env index 2f20824..0d0885a 100644 --- a/scripts/frank-archon.env +++ b/scripts/frank-archon.env @@ -1,306 +1,99 @@ -# frank-archon.env — radiance (StillDeadcode, Apache-2.0) serving the -# PUBLISHER's 4-bit-codebook container of the official Qwen/Qwen3.8-Flash-Next -# (MODEL_FILE below, fetched by scripts/download-frank-archon-iq4r.sh) on -# BOTH R9700s (TP=2), dankserver :7770. -# 2026-10-15 PAYLOAD SWAP: the model moved from the in-house w4dn8 conversion -# of the Swift1.5 finetune to the engine author's published container -# qwen3.8-next-flash-fp8-iq4r-moe.rad (116,322 MiB, sha256-verified at fetch — -# the w4dn8 measured 116,416 MiB, so this shape carries over with 94 MiB to -# spare). NO argument changed: every number below is the 2026-10-13 shape, -# kept verbatim. The swap commit is the rollback (the w4dn8 payload stays on -# disk). - +# frank-archon.env — radiance (StillDeadcode) on BOTH R9700s (TP=2), +# Dankserver :7770, publisher's 4-bit-codebook container of the official +# Qwen3.8-Flash-Next. # -# SHAPE (2026-10-13): THREE decode slots x 200,000 tokens, fp8 (quantized) KV, -# with the per-card budget spent on resident experts: 25,600 MiB of slab + -# 5,120 MiB of pool = 30,720 MiB, inside the 30,969 MiB ceiling the engine's own -# budget table allows (card 32,624 MiB, settle floor 209 MiB). The pool is sized -# to the contract and nothing more (3 x 200,000 needs 5,063 MiB at fp8; 5,120 -# predicts 608,027 tokens, 1.3 % slack), because every MiB taken from the slab -# is prefill and TTFT (2026-10-11 A/B: slab -14.8 % cost prefill -29 %, TTFT -# +25 % at the full window) while KV beyond the contract buys nothing. -# Consequence, deliberate: a FOURTH full-height conversation does not fit this -# pair, and the engine refuses it — it never truncates. See the KV block for the -# arithmetic and the measured dtype law. The w4dn8 payload is the swap -# rollback and keeps its trio under the w4dn8 runbook; the fp8-native payload -# that came before it was DELETED from disk 2026-10-15 (Frank's call, to free -# the volume for this download — rebuildable via the fetch + build scripts -# named in the fp8-native runbook). -# -# 2026-10-05 cutover from FreeToken (PR #587 ROCm, one card, experts in -# pinned host RAM; git tag frank-archon-freetoken-pr587-20261005 is the -# pre-cutover state — its unit/env/serve trio and Dockerfile are all in the -# tag and rebuildable). Before that: tcclaviger vLLM fork (tag -# frank-archon-vllm-shapeG-20261003). -# -# WHY RADIANCE: the engine is C++/HIP with libr4d kernels built for -# gfx1201, drives `podman run` directly from the digest-pinned -# stilldeadcode/radiance image (287 MiB; host needs only the amdgpu driver), -# and its qwen4exp plugin reads the modelopt FP8 encoding NATIVELY -# (`X.weight_scale_inv` is the scale plane — checkpoint.h). The PLE n-gram -# table (51B params, 47.7 GiB) is served from pinned host RAM via io_uring -# reads — Frank's call: "Native FP8. Experts from ram. Pin the PLE to ram." -# THE 4-bit-codebook publisher shape is the DEPLOYED payload since -# 2026-10-15. The card's own run line (--host-pool-mib 12288, -# --expert-vs-cache-ratio 0.82, seqs 8, spec 3) is the publisher's -# suggestion on 32 GB cards; this file's shape is the fleet's measured one. -# -# PAYLOAD: /home/dank/models/rad/qwen3.8-next-flash-fp8-iq4r-moe.rad — the -# official Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's -# own author (recipe qwen4exp-w4nl64-i8-hc8m: experts as 4-bit codes into a -# 16-level non-uniform codebook after a 128-point Walsh–Hadamard rotation, -# GPTQ + 3 coordinate-descent sweeps on a 10M-token calibration, ten -# bf16-protected experts per layer, int8 W8A8 trunk + lm_head, fp8 HC -# mixtures, PLE n-gram fp8 fixed-scale, MTP head, bf16 vision tower; card KL -# 0.0789 all-tokens / top-1 92.85 % vs bf16). Fetched byte-exact + sha256 by -# scripts/download-frank-archon-iq4r.sh; the recipe/LICENSE stay in the -# staging dir beside it as provenance. ROLLBACK: revert the swap commit — the -# in-house qwen4exp-swift15-w4dn8-i8-hc8m.rad (scripts/convert-swift15-w4dn8.sh, -# KLD-gated) stays on disk. Behind it, the fp8-native -# qwen4exp-fp8-native.rad was DELETED from disk 2026-10-15 (Frank's call, -# one-time rm, to free the volume for this payload); rebuild = -# scripts/download-frank-archon.sh + scripts/build-frank-archon-rad.sh — a -# NO-RECIPE rad-convert copy of the official 131-shard checkpoint under the -# descriptor law (no shipped plugin claims (qwen4exp, "fp8"), so the staging -# config.json drops quantization_config; per-weight encodings are data, not -# the descriptor). -# -# SERVING FACTS (docs/GUIDE.md, docs/DOCKER.md, compose/flashnext.yaml): -# - The HTTP listener binds AFTER the full load (main.cpp: engine.start() -# then engine.serve()) — during load the port refuses connections, so a -# bare curl /health gate is honest. /health == /ping: 200 JSON while -# serving, 503 only while stopping (core/server/admin.cpp:385). -# - Reasoning: --reasoning-format defaults to auto, emitting thinking in -# `reasoning_content` (config.cpp). Tool calls: the qwen3.8 dialect, from -# the chat format. Both need no parser flags. -# - Sampling: the OpenAI endpoint accepts temperature, top_p, top_k, min_p, -# typical_p, presence_penalty, frequency_penalty, repetition_penalty, -# repeat_last_n, chat_template_kwargs, reasoning_effort, -# enable_thinking, preserve_thinking (core/server/oai.cpp:194-205) — -# models.yml's extraBody set is fully inside it. Unknown params are -# refused (400), which is why nothing unverified rides the wire. -# - MTP: the .rad carries the model's mtp.* head (the publisher's recipe -# names the mtp.fc_hidden/fc_embedding/draft_head rules; card default -# depth 3). Served at the depth named by FRANK_ARCHON_SPEC_TOKENS below — -# the depth this shape's decode ladder was measured at. -# - The fused TP all-reduce is declared over max-num-seqs x -# (1 + --num-speculative-tokens) rows; 3 x (1+2) = 9 rows, under the 25-row -# exactness bound of the wht6 wire. -# Engine pin: Codeberg tag v1.0.13 (commit 53282fa), Docker Hub tag 1.0.13, -# index digest below. REJECTED UPGRADE: 1.2.0 (sha256:665775ed1d90de6dbb -# 25558e07fc60d20328f1346f96f3c6e00845eb763f01fa, tag 1.2.0, commit -# e95d397) was benched 2026-10-06 on this exact shape (fp8-native rad, -# vram pair, spec 2, seqs 3, 43.6k greedy probe): decode 29.21 t/s and -# draft acceptance 0.7123 vs the 1.0.13 control's 30.08 and 0.7549. -# Acceptance on the greedy harness is deterministic (the control -# reproduced the recorded 0.755 exactly), so the drop is a real numerics -# change — the 1.2.0 MoE weight-rotation change (92b8aaf) is the named -# suspect; decode loss tracks the acceptance ratio (0.943 vs 0.945). -# The vram-pair translation itself is exact (1.2.0 boot ledger took -# vram_weights 25.46 GiB / vram_kv 3.46 GiB per card byte-identical to -# the 1.0.13 ledger at EVC 0.82). Re-test 1.2.0+ when the rotation change -# is untangled; 1.2.0 also REMOVED --expert-vs-cache-ratio in favour of -# the --vram-weights-mib / --vram-kv-mib pair (core/config.cpp) — this -# file already uses the pair, which BOTH pins accept (rollback symmetry). +# ONE OWNER: shape, payload identity, eras, laws and every measured number live +# in serverAppConfigs-docs/LLM/frank-archon.md. This file carries the knobs plus +# only the reason a reader editing THIS line cannot recover from the page. +# Where the two disagree, this file wins on values; the boot ledger arbitrates +# the engine shape. FRANK_ARCHON_IMAGE=stilldeadcode/radiance@sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e -# The .rad container FILE (mounted under /models) and the state dir for the -# prefix-cache disk tier (mounted rw at /data). The model dir holds ONLY -# .rad files; the raw HF download and its staging dir live beside it and -# are never mounted. +# 1.0.13 is pinned by digest on purpose: 1.2.0 is a DEAD END on this payload +# (acceptance 0.7123 vs 0.7549 — the page's Dead ends section names the suspect). +# The .rad file mounted under /models; /data holds the prefix-cache disk tier. +# The model dir holds ONLY .rad files — the HF download and staging stay beside it. FRANK_ARCHON_MODEL_DIR=/home/dank/models/rad FRANK_ARCHON_MODEL_FILE=qwen3.8-next-flash-fp8-iq4r-moe.rad FRANK_ARCHON_STATE_DIR=/home/dank/radiance-state -# The state dir's tune/ subdirectory holds rad-tune's kernel-selection -# cache (the engine's selector reads $RADIANCE_HOME/tune/.tune -# at boot; $RADIANCE_HOME defaults to /opt/radiance, in the image, so the -# cache must ride a host mount — serve script mounts it, see the runbook's -# "Tuning levers"). -# Container identity is the SERVICE name (Frank's call, 2026-10-05): one -# name across systemctl and podman ps. The runtime changes; the name does -# not. The legacy list retires every predecessor identity that could hold -# the GPUs or the port. +# state dir's tune/ subdir carries rad-tune's kernel-selection cache; the engine +# reads $RADIANCE_HOME/tune/.tune and that defaults to /opt/radiance +# (inside the image), so the cache only survives on a host mount. +# Container identity is the SERVICE name: one name across systemctl and +# `podman ps`; the runtime changes, the name does not. The legacy list retires +# every predecessor identity that could hold the GPUs or the port. FRANK_ARCHON_CONTAINER=frank-archon FRANK_ARCHON_LEGACY_CONTAINERS="frank-archon-qwen frank-archon-r9v qwen-fp8 glm-probe" -# Served model id — Protoss name (wire rename 2026-10-09). The container, -# unit, scripts and this file keep the frank-archon name above; only the -# alias on the wire moved. Radiance validates the requested model name, so -# every client (harness, bench scripts, probes) MUST send this value. +# Served id. Radiance VALIDATES the requested model name, so every client must +# send exactly this value (unlike llama.cpp and halogen, which ignore it). FRANK_ARCHON_SERVED_NAME=frank-archon FRANK_ARCHON_PORT=7770 -# Frank's chat-template call: froggeric's fixed Qwen3.8 template (the -# repo's root chat_template.jinja, vendored here; the checkpoint's own -# template is baked into the .rad by rad-convert and this overrides it). -# Radiance derives the reasoning/tool-call reply format from it. +# Frank's chat-template call: froggeric's fixed Qwen3.8 template, vendored in +# the repo; it overrides the template baked into the .rad, and radiance derives +# the reasoning and tool-call reply format from it. FRANK_ARCHON_TEMPLATE=/home/dank/serverAppConfigs/scripts/qwen38-froggeric-v22.5.jinja # --- engine shape (the boot ledger arbitrates every number below) --------- -# Tensor parallel: BOTH R9700s. The board swap (2026-09-20, ProArt B850 -# x8/x8 on root ports) made collectives legal on this host; radiance's TP -# wire is libr4d's own, not RCCL. +# TP=2 needs the 2026-09-20 board (x8/x8 on root ports): a collective cannot be +# dispatched on a card whose PCIe chain lacks atomics. Radiance's wire is +# libr4d's own, not RCCL. FRANK_ARCHON_TP=2 -# Lossy WHT 6-bit wire for large all-reduces (publisher's production choice: -# +5.6-6.3% prefill, ~-1.3% 8-stream decode at TP2). min-kb stays the -# engine default. +# Lossy WHT 6-bit wire for large all-reduces (publisher's production choice). It +# stays EXACT for single-sequence decode and any step of <= 25 rows — see +# MAX_SEQS x (1 + SPEC_TOKENS) below. FRANK_ARCHON_TP_WIRE=wht6 -# Decode slots: THREE, so the served window is a CAPACITY CONTRACT — three -# concurrent conversations of --max-model-len fit the KV pool (see the KV -# block below). The "seqs 2 cannot boot" law is NOT radiance's: it was the -# tcclaviger vLLM fork's torch.compile shape specialization on -# query_start_loc (ConstraintViolationError, measured 2026-10-02, retired -# with that runtime). Radiance booted seqs 2 at the fp8-native first boot, -# and the 2026-10-06 decode sweep measured seqs 2 and 3 statistically tied -# (30.9-33.1 t/s, inside the +-3% noise floor) — slot count buys concurrency, -# not speed, so a slot costs KV, never decode. -# Cost of each extra slot: 110 MiB of per-slot recurrent state inside the KV -# pool, and the fused TP all-reduce is declared over seqs x (1 + spec) rows — -# 3 x 3 = 9 here, under the 25-row bound at which the wht6 wire stays EXACT -# (single-sequence decode and any step of <= 25 rows is bit-exact; only busier -# steps are lossy). -# The pool is sized to this contract and nothing more (see the KV block): -# 3 x 200,000 at fp8 = 5,063 MiB, pool 5,120, and the freed 1,792 MiB went into -# the resident expert slab. The trade is explicit and it is Frank's call: -# a FOURTH full-height conversation no longer fits (5,120 MiB of pool backs -# ~608k tokens; 4 x 200,000 needs 6,601), so a fourth 200k request is refused -# as too large to fit — it is NOT squeezed, paged, or truncated. Third-party -# traffic that wants four full-height sessions is a --vram-kv-mib change, not -# a code change. +# Decode slots. A slot costs KV, never decode; the pool is sized to this +# contract and nothing more, so a FOURTH full-height conversation is REFUSED as +# too large to fit — never squeezed, paged or truncated. That is Frank's +# explicit trade: the freed MiB went to the expert slab. FRANK_ARCHON_MAX_SEQS=3 -# Served window. Native is 262,144; 200,000 is the publisher's production -# value (compose) and what models.yml's contextWindow must quote — a longer -# prompt is refused by the engine, never truncated. The engine treats -# `kv group tokens >= max-model-len` as a HARD limit ("ONE request longer -# than N tokens is accepted by the API and then refused as too large to ever -# fit — this is a hard limit, not a backing ratio"), so the pool is sized -# for seqs x maxlen, not for one sequence. +# Served window. The engine treats `kv group tokens >= max-model-len` as a HARD +# limit, and models.yml contextWindow MUST quote this number (over-length is a +# refusal, not a truncation). The pool is sized for seqs x maxlen, not one +# sequence. FRANK_ARCHON_MAX_MODEL_LEN=200000 FRANK_ARCHON_PLACEMENT=expert_tiered -# Pinned host pool for the expert rows that do not fit VRAM. vram_weights and -# this pool serve the per-rank resident set TOGETHER: at the fp8-KV pair -# (26071/3543) the ledger showed the VRAM slab filled to 259 KiB of slack, -# i.e. the pair was exactly sized. Cutting vram-weights by 3.8 GiB to pay for -# the bf16 KV pool SPILLS ~3.8 GiB/rank here, so the pool moves with it — -# under-sizing it is a boot failure (36864 refused the boot in the fp8 era). -# The boot ledger's coverage line remains the referee, not this arithmetic. +# Pinned host pool for expert rows that do not fit VRAM. It moves WITH +# VRAM_WEIGHTS: the pair and this pool serve the per-rank resident set together, +# so cutting the slab spills here — under-sizing it is a boot failure (36,864 +# refused a boot). The ledger's coverage line is the referee, not this file. FRANK_ARCHON_HOST_POOL_MIB=44928 -# VRAM left unclaimed per card (publisher default 96 MiB). +# VRAM left unclaimed per card (publisher default 96 MiB); the engine's settle +# floor is built from it. FRANK_ARCHON_GPU_HEADROOM_MIB=96 -# VRAM split of claimed memory per card, MiB (0 = derive from the card). -# THE CEILING, read off this engine's own VRAM-budget table (boot 2026-10-13, -# pair 24000/5120): card 32,624 MiB; budgets claimed 29,120; `3.42 GiB is -# neither budgeted nor accounted — the driver's context, the activation arena -# and --gpu-headroom-mib come out of it; anything past those is throughput on -# the floor`; and `card 0 settled at 2.01 GiB free against 209.19 MiB (the -# 96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the static region)`. -# So the reserve above the ENGINE's floor is 2,058 - 209 = 1,849 MiB and the -# ceiling is 29,120 + 1,849 = 30,969 MiB. The 2026-10-11 bound of 29,120 -# ("31,856 - 1,710 outside both budgets - keep 1 GiB free") was MY reserve, not -# the engine's; the pair below spends the difference and lands 458 MiB above -# the engine's floor. `weights + kv <= 30,969` is the gate. -# THE PRICE, and it is paid twice: on this engine idle KV VRAM is ALSO the -# expert cache's spillover (`mover: 2275 slots can be lent out of the KV -# cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <= 256 tokens`), -# so a bigger KV pool steals weight residency from the slab AND from the -# lendable cache. Priors on this card/model: stealing 2.14 GiB of weights for -# a bigger pool (MAX_MODEL_LEN 262144) cost 2.5% decode; the EVC ladder's -# other direction (0.82 -> 0.84, +2.5 GiB weights) cost 8.9%. The 2026-10-11 -# exact-KV A/B measured the slab's own price (22208 vs the fp8-era 26071): -# prefill -29 %, TTFT at the full window +25 %, decode inside the noise band on -# 2 of 3 passes, draft acceptance UP 0.661 -> 0.740. -# 2026-10-13 (late): THREE slots, so the pool is cut to the contract (5,120) -# and every spare MiB goes to the slab: 22208 -> 24000 (the third slot's -# surplus KV), then 24000 -> 25600 (the card's settling slack). The direction -# the A/B data points: prefill is slab-limited, KV beyond the contract is not. -# The margin is thin by design (608,027 predicted tokens against the 600,000 -# contract), so the boot ledger's `backs N sequences all at 200000 tokens at -# once` line is the referee for THIS pair and the concurrent probe is its -# regression test. -# VERDICT on the 25,600 arm (measured 2026-10-13, scripts/probe-frank-archon-kv.py): -# the threshold claim is FALSIFIED — the slab is a SLOPE at this end. A-cold -# 43.5k single-stream: 22,208 -> 3,397, 24,000 -> 3,290, 25,600 -> 3,584 t/s; -# the 3-way 197k arm's aggregate prefill 3,051 -> 3,589 t/s (wall -# 193.7 -> 164.5 s). So +1,600 MiB of slab = +8.9 % single-stream and -# +17.6 % concurrent, both outside the +-3 % noise floor. The 22,208/24,000 -# pair landed inside noise of EACH OTHER, which is what made it read as a -# plateau; two points are not a curve. -# The cliff that remains is between 25,600 (3,584) and 26,071 (4,379, the -# 2026-10-11 fp8 control at seqs 3 / pool 3,543) — +22 % for +471 MiB. At THREE -# slots that is only reachable by breaking the ceiling (26,112 + 5,120 = 31,232 -# vs 30,969, ~214 MiB above the engine's floor); at TWO slots it is free: -# pool 3,072 + slab 27,897 = exactly the ceiling, slab above the cliff. +# ONE budget per card: VRAM_WEIGHTS + VRAM_KV <= 30,969 MiB, the ceiling the +# engine's own budget table prints (derivation and the slab/pool price: the +# page's VRAM laws). Idle KV VRAM is ALSO the expert cache's spillover, so a +# bigger pool pays twice — which is why the pool sits at the contract and the +# slab takes the rest. FRANK_ARCHON_VRAM_WEIGHTS_MIB=25600 FRANK_ARCHON_VRAM_KV_MIB=5120 -# KV cache QUANTIZED (fp8). This engine's flag takes ONLY bf16|fp8 — there is -# ONE kv_dtype for the whole cache, so "exact K, quantized V" (llama.cpp's -# --cache-type-k/--cache-type-v pair, which is what frank-zealot runs) is -# UNSERVABLE here: the rung below bf16 is fp8 on both halves. -# Measured per-rank cost, read out of the boot ledger's group table (fp8 boot -# 2026-10-08, bf16 boot 2026-10-11): -# kv_attn 13 layers, 4-token blocks, page 26.00 KiB (fp8) -> 52.00 KiB -# kv_qsa_bkey 13 layers, 4-token blocks, page 3.25 KiB at BOTH dtypes — -# the QSA block key is an index, not attention state, and does -# NOT follow --kv-cache-dtype -# => 7.3125 KiB/token fp8, 13.8125 KiB/token bf16: bf16 costs 1.89x fp8, -# not 2x, precisely because the bkey plane is dtype-fixed. -# The dtype-INDEPENDENT terms also live in the pool: kv_gdn_state 54 MiB x 2 -# copies per sequence (+ conv/tail/ple ~= 110 MiB a sequence) and the prefix -# checkpoint region 8 x 55.99 = 448 MiB. So: -# pool_mib >= tokens x 7.3125 KiB + 110 x seqs + 448 -# 3 x 200,000 at fp8 needs 4,285 + 330 + 448 = 5,063 MiB; the pool is 5,120, -# and it predicts (5120 - 330 - 448) x 1024 / 7.3125 = 608,027 tokens: the -# three-slot contract with 1.3 % slack, every MiB above 5,063 having been -# handed to the expert slab instead. The ledger line is the referee ("backs N -# sequences all at 200000 tokens at once") — a pair this tight is read off the -# boot, never assumed. -# At bf16 the same three slots cost 8,093 + 778 = 8,871 MiB, which would cut -# the slab to 20,249 MiB (-9 %); at FOUR slots bf16 costs 10,791 + 888 = -# 11,679 MiB and the slab falls to 17,441 (-33 %). fp8 keeps 22,208 at both -# slot counts. The 2026-10-11 bf16 shape (6,912 / seqs 2) backed 462,752 -# tokens = 2 x 200,000; the historical fp8 pool of 3,543 gave 386,904 (1.93 -# sessions), and the four-slot fp8 boot of 2026-10-13 19:21 backed 843,276. -# Secondary saving: the prefix-cache entry shrinks with the attention plane -# (one entry 110.50 KiB at bf16 -> 58.50 KiB at fp8, measured on the two boot -# ledgers' `kv tier transfer` lines), so the host tier below parks twice as -# many conversations as it did at bf16 — no re-size needed to park three slots. +# fp8 is the ONLY rung below bf16 on this engine: one kv_dtype for the whole +# cache, so llama.cpp's exact-K/quantized-V split (frank-zealot) is unservable +# here. bf16 costs 1.89x fp8, not 2x — the QSA block-key plane is dtype-fixed. FRANK_ARCHON_KV_DTYPE=fp8 -# Spec decode: the checkpoint's own MTP head, served at depth 2 — the -# publisher's card ships depth 3 as its peak; this fleet's ladder was -# measured at 2 and the swap keeps every argument, so 2 stays (rows -# 3 x (1+2) = 9, inside the wht6 25-row exactness bound). +# The checkpoint's own MTP head at depth 2: this fleet's ladder was measured at +# 2, and 3 x (1+2) = 9 rows stays inside the wht6 25-row exactness bound. FRANK_ARCHON_SPEC_TOKENS=2 FRANK_ARCHON_MAX_BATCHED_TOKENS=2048 -# Prefix cache tiers (publisher pattern). The host tier is sized in ENTRIES, -# and one entry tracks the cache dtype (58.50 KiB at fp8, 117 KiB at bf16, per -# the ledger's `kv tier transfer` line): at fp8, 8192 MiB parks ~5.8 sessions -# of 200k (at bf16 the same tier parked ~2.9), so four live slots plus a full -# parked set fits with the tier as it stands — at RAM the host has (104 GiB -# available with the service loaded). +# Prefix cache tiers. The host tier is sized in ENTRIES and one entry tracks the +# cache dtype (58.50 KiB at fp8), so a dtype change re-sizes the parked set +# without touching these numbers. FRANK_ARCHON_PREFIX_HOST_MIB=8192 FRANK_ARCHON_PREFIX_DISK_MIB=65536 -# Frank's pin law: the PLE n-gram table (47.7 GiB) serves from pinned host -# RAM the cards read. `ram` REFUSES the start if the memlock limit is too -# small — the unit's LimitMEMLOCK=infinity is the other half of this law. +# Frank's pin law: the PLE n-gram table serves from pinned host RAM the cards +# read. `ram` REFUSES the start if memlock is too small — the unit's +# LimitMEMLOCK=infinity is the other half of this law. FRANK_ARCHON_NGRAM_PLACEMENT=ram -# TTM preflight: the pinned ledger (2 x 44 GiB pools + 47.7 GiB PLE + 8 GiB -# prefix host = ~147 GiB) rides TTM's shared pages, capped by ttm's -# pages_limit (the current kernel exposes NO pages_max; the cap function -# takes the min over the knobs that exist). Below the plan the load dies -# mid-pin with an allocator error instead of this named refusal. -# THE CAP IS A RUNTIME KNOB — the kernel default is half of RAM, so every -# raise dies with a reboot. Measured the hard way: the 2026-10-05 manual raise -# to 52,428,800 pages (200 GiB) was gone after the 2026-10-12 23:17 POWER -# FAILURE, the cap fell back to 125,988 MiB, and the preflight refused the -# start 6,919 times in ~20 h while Restart=always re-ran it. The gate in -# scripts/ttm-preflight.sh now RAISES the knob itself before refusing (the -# root unit is privileged; the knob is a 0644 root module parameter), so the -# recovery is a boot, not a visit to the machine. +# TTM preflight. The pinned ledger (~147 GiB) rides TTM's shared pages, and the +# kernel RESETS pages_limit to half of RAM on every boot; below the plan the +# load dies mid-pin with an anonymous allocator error. ttm-preflight.sh raises +# the knob itself before refusing by name, so recovery is a boot, not a visit. FRANK_ARCHON_TTM_MIN_MIB=150000 FRANK_ARCHON_TTM_TARGET_MIB=204800 -# Escape hatch for boot-ladder experiments. Unset -> no extra argv. Keep -# multi-flag values as one string; the serve script expands it unquoted on -# purpose. +# Escape hatch for experiments. Unset -> no extra argv; the serve script expands +# this unquoted on purpose, so keep multi-flag values as one string. FRANK_ARCHON_EXTRA_ARGS= -# The .rad's minimum size in MiB (deployer gate; a truncated fetch is a -# guaranteed long-timeout boot instead of a fast failure). The deployed -# publisher container measures 116,322 MiB (121,969,901,568 B); the -# in-house w4dn8 rollback measured 116,416 MiB. Floor sits just under both. +# Deployer gate: a truncated fetch must be a fast refusal, not a long-timeout +# boot. Floor sits under the live payload and under every rollback payload +# (see the page's Eras table for what is on disk and what needs a re-download). FRANK_ARCHON_RAD_MIN_MIB=110000 diff --git a/scripts/frank-dragoon.env b/scripts/frank-dragoon.env index 5d8a0cc..045a770 100644 --- a/scripts/frank-dragoon.env +++ b/scripts/frank-dragoon.env @@ -1,209 +1,82 @@ # frank-dragoon.env — tunables for the frank-dragoon endpoint on Dankclaw's # Strix Halo iGPU (gfx1151), host port 7770. # -# RUNTIME (2026-10-17, Frank): halogen-flash-server 0.16.2 — the -# closed-source C++/HIP engine (image ghcr.io/peonist-ai/halogen-flash-server, -# digest-pinned below) serving Quat3rnion/halogen-qwen3.8-flash-next-v2- -# abliterated — the NATIVE v2 conversion (2026-10-11) of orcarouter's pinned -# BF16 Qwen3.8-Flash-Next, abliterated — as a podman container under a root -# unit, driven by scripts/serve-frank-dragoon.sh. This CUTS OVER the -# 2026-10-06 Swift1.5-finetune shape; rollback = revert the swap commit and -# redeploy: the swift15 payload stays on disk at -# /home/dank/models/halogen-flashnext-v2/, the 27B-era payload beside it. +# ONE OWNER: payload identity, shape, eras, laws and every measured number live +# in serverAppConfigs-docs/LLM/frank-dragoon.md. This file carries the knobs +# plus only the reason a reader editing THIS line cannot recover from the page. +# Where the two disagree, this file wins on values; the boot ledger arbitrates +# the memory figures. # -# MEASURED A/B, both boots on this box, identical battery traffic (engine -# tools with the service stopped, behavior probes against it, 2026-10-17): -# NIAH 18/18 vs 17/18, refusal census 0/5 on both, tool calls and the -# red-PNG vision probe identical, needle hit at both depths of a 21.5k-token -# haystack, decode 50.0 vs 46.8 t/s, cold prefill 1,604 vs 1,425 t/s, -# wikitext-2 PPL 3.968 vs 3.877 (the finetune's edge is gone: +2.3 %) and -# thinking tokens 1,476 vs 784 mean — the finetune compressed reasoning, the -# base thinks ~1.9x longer per answer. -# -# The A/B ran at the card's ctx/pool of 262,144. Afterwards Frank lowered ctx to -# 200,704 (RAM + the harness's compaction budget) and, on 2026-10-18, raised the -# pool to 2 x ctx = 401,408 so BOTH slots can carry a full-length request at -# once; the rates above are the ceiling-262,144 record and the production ledger -# is the ctx-200,704 one. -# -# WHY THE RUNTIME MOVED WITH THE MODEL: the payload is .hgn — per its own -# card it "will not load in transformers, vLLM, or llama.cpp" — and the v2 -# .hgn loads ONLY in the flash-server line (the old halogen engine is for -# the 27B p1w4d file; the v2 quickstart pins flash-server). What the move -# buys (card + live probe evidence): MTP spec decode (53-57 t/s greedy, -# 3-10x serial), ~1,640 t/s prefill at 34k, ~72% fewer thinking tokens than -# the censored 27B (905 mean), the ABLITERATED base (refusals should NOT -# return), IMAGE INPUT (the v2 vision tower loads beside the checkpoint — -# red-PNG probe verified on the card; the 27B engine ignored its encoder), -# tool calls verified, and the native 262,144 context — served at 200,704 by -# choice from 2026-10-17 (see SHAPE: the ceiling is a RAM decision, not a -# capability limit). -# -# SHAPE — the knobs that are NOT free, with the arithmetic: -# HALOGEN_CTX=200704 is the PER-REQUEST ceiling and the per-conversation -# hard limit — a prompt over it is a 400 naming the limit, never a silent -# truncation. LOWERED from 262,144 on 2026-10-17 at Frank's direction (the -# checkpoint's native window is 262,144 and the engine serves anything -# below it): every position costs KV pool RAM and shrinks the prompt-cache -# entry (111 MiB at 262,144, ~85 MiB here). The 400-zone law is INVARIANT -# to the value — it says the harness window must sit ~45k UNDER the engine -# ceiling, because the harness clamps max_tokens to contextWindow − est − -# 4096, its chars/4 est UNDER-counts the rendered prompt by ~3 % on -# code/agent traffic, and compaction only fires at 85 % (measured 2026-10-07 -# at the old 200,192 ceiling: est 152,938, server 157,388, over by 354). -# Harness contract (models.yml): contextWindow 155648 (= ceiling − 45056, -# the gap that absorbs the estimate error), maxTokens 45056. -# Compaction fires at 85 % = 132,301 est ≈ 136k measured — the 180k target -# of the 262,144 era is unreachable at this ceiling: the window itself is -# 155,648. Worst-case request: prompt + budget <= 155648 − 4096 + 3 % -# est-error ≈ 156.2k <= 200,704 — ~9.5× the 3 % error of margin (robust to a -# ~25 % under-count). Compaction rides on frank-zealot: input ≈ threshold − -# keepRecent(20k) ≈ 112k, inside its 200,704 slot. -# HALOGEN_KV_POOL_POSITIONS=401408 = SLOTS x CTX (2 x 200,704), the top of the -# legal band (the seam enforces ctx <= pool <= slots * ctx). Still ONE pool -# behind HALOGEN_KV_SLOTS=2 (ledger form: "2 slots over ONE -position KV -# pool, each request up to HALOGEN_CTX"), but sized so BOTH slots can hold a -# MAXIMUM-length request at the same time — the parity with frank-zealot's -# 401,408-position pool that Frank ordered on 2026-10-18. The PER-REQUEST -# ceiling does NOT move with it: HALOGEN_CTX stays 200,704, because the -# checkpoint's native window is 262,144 and the engine refuses to go past it -# without HALOGEN_ROPE_YARN (its own 400 text says so). ~28 KiB/position -# (FLAGS.md) ⇒ the DEPLOYED 2026-10-18 boot RESERVES 11.0 GiB (`reserving the -# KV pool, 401408 positions (about 11.0 GiB)`), where pool == ctx (200,704) -# cost 5.5, 262,144 cost 7.2 and the 2026-10-06 shape's 524,288 measured 14.6. -# The preflight formula bills 11.3 (block scratch + ~115 MiB/slot of O(1) -# state) — quote the RESERVATION. FLAGS.md law: "RAM the pool -# takes is RAM that table loses" — the 47.68 GiB n-gram table surrenders this -# much page cache, so the warm prompt-cache ratio is what to re-measure. -# HALOGEN_KV_SLOTS=2 restores the w4b-era shape on this box: the MTP -# drafter is NOT slot-locked on this engine (batching and speculation -# coexist — DFlash-era slots>1 law does NOT carry over; the card's own -# quickstart ships 2 slots WITH the head), so the second slot is -# concurrent capacity, not a speculation tax. -# Memory ledger against the box's 121.2 GiB unified (the n-gram table is -# mmap'd page cache beside the checkpoint, not a pinned charge — the boot -# ledger agrees: "mapped 51.2 GB, NOT pinned (Pin::None)"): -# weights 62.9 GiB locked (0.8 of them the vision tower) + KV 11.0 + -# working 5.7 (HALOGEN_MAX_TOK=8192) = 79.6 GiB engine-side (the boot -# ledger of the 401408-position deploy, 2026-10-18), with the 1.3 GiB -# prompt cache (12 entries x 111 MiB at ctx 200704) booked separately by -# the ledger; HALOGEN_HOST_RESERVE_GIB=20 keeps the host honest. Against -# the 121.2 GiB box that leaves ~20 GiB over the reserve (MemAvailable -# read 22 GiB right after the raise) for baseline co-tenancy -# (frank-zealot, ComfyUI, Caddy/observ ~10-15). The boot ledger is the -# referee — read it after deploy and re-size if the engine prints a -# clamp. -# HALOGEN_SPEC_ADAPT=0 is MANDATORY (card): the default adaptive policy -# turns the MTP head off for whole requests after low acceptance and -# HALVES decode as a result. Frank accepted the head always-on trade -# (temp 1.0 traffic costs acceptance; the card still ships the recipe). -# -# CLIENT CONTRACT deltas vs the 27B shape (harness side lives in -# ~/.omp/agent/models.yml): -# * SERVER-SIDE SAMPLER EXISTS NOW: HALOGEN_TEMPERATURE/TOP_P/TOP_K set -# the card's thinking recipe (1.0 / 0.95 / 20) as engine defaults, so -# an absent temperature is no longer greedy. models.yml extraBody -# carries the same values — consistent, not load-bearing. -# * HALOGEN_MAX_TOK=8192 is only the DEFAULT for requests that omit -# max_tokens; the harness sends at most its maxTokens 45056. Requests -# over HALOGEN_MAX_TOKENS_CAP (65536) are a 400 naming the cap. Cap and -# queue timeout are COUPLED (FLAGS.md table): 65536 at ~10 t/s serial -# worst case needs > 6550 s; 7200 honors it. Do not move one alone. -# * max_tokens INCLUDES thinking; a reasoning reply over a small budget -# returns empty content with finish_reason "length". -# * Prompt cache is ON by default (PROMPT_CACHE=2): a follow-up turn at -# 100k context re-prefills ~2 s instead of ~88 s cold. No pin needed — -# unlike the 27B engine, auto-sizing fits this box's ledger. -# * Thinking lands in reasoning_content (same as the 27B engine); the -# v2 chat template rejects reasoning_effort outside xhigh|medium|low, -# so the models.yml fold (high->xhigh) still applies. -# * Vision: OpenAI image_url content parts only (the fleet's standard -# contract). >=256 px probe images or the model answers "video". -# * Concurrency at depth is tight now: the KV pool is ONE 200,704-position -# budget behind 2 slots — exactly one maximum-length request fills it. Two -# conversations coexist only while they stay under ~100k each. The -# PER-REQUEST ceiling moved WITH the pool (262144 -> 200704), so the harness -# contract moves too: contextWindow 155648 = ceiling − 45k, never higher. -# * response_format json_object/json_schema unsupported in this line -# (no constrained decoding) — unchanged from the 27B engine. -# * The engine's token protocol has no auth and binds loopback INSIDE -# the container; only the frontend port (8731) is published. Same as -# before. Kernel needs CONFIG_HSA_AMD_SVM (CachyOS ships it) and -# amdgpu.noretry=0 (check at deploy). - -# Image: digest-pinned (fetch law: tag pulls silently re-point; the digest -# is what was qualified). 0.16.2 = the version the model card names as its -# quickstart pin; RepoDigest captured at pull on 2026-10-06. +# The .hgn format is the lock: it loads ONLY in the flash-server line, so the +# engine moves with the payload, never ahead of it. +# Image digest-pinned (a tag pull silently re-points; the digest is what was +# qualified — 0.16.2 is the version the model card names as its quickstart pin). FRANK_DRAGOON_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a -# Container identity: kept from the 27B era (unit ExecStop names it). +# Container identity kept from the 27B era: the unit's ExecStop names it. FRANK_DRAGOON_CONTAINER=frank-dragoon-halogen -# Old quadlet-era container identity; the serve script clears stale names. +# Predecessor identity the serve script clears if it is still holding the GPUs. FRANK_DRAGOON_LEGACY_CONTAINERS=frank-dragoon -# host:container — the frontend listens on 8731 inside (HALOGEN_API_PORT -# default); the engine port (8730) is never published. +# host:container — the frontend listens on 8731 inside; the engine's own port +# (8730) has no auth and is NEVER published. FRANK_DRAGOON_PUBLISH=7770:8731 FRANK_DRAGOON_MODEL_DIR=/home/dank/models/halogen-v2-abliterated -# The checkpoint: Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated -# qwen38-flash-next-v2-abliterated.hgn — the NATIVE 2026-10-11 v2 conversion -# of orcarouter's pinned BF16 Qwen3.8-Flash-Next, abliterated. NOT the writer -# transplant this filename used to carry: 66,687,678,432 B where the swift15 -# r2 measured 66,687,678,464 B — same filename family, different artifact. -# Hub LFS sha256 verified at fetch by scripts/download-frank-dragoon-v2abl.sh. +# Same filename family, DIFFERENT artifact: this is the native v2 conversion, +# 32 bytes smaller than the writer transplant that used to carry the name. +# Never infer the payload from the filename — this size and sha256 are the gate. FRANK_DRAGOON_CHECKPOINT=qwen38-flash-next-v2-abliterated.hgn FRANK_DRAGOON_CHECKPOINT_BYTES=66687678432 FRANK_DRAGOON_CHECKPOINT_SHA256=b5bd925fa2b44343be4a2cd328cae6b57ce7ffa2fe2cc56281717377e5a1ce8a -# The n-gram lookup table: REQUIRED (the engine reads it beside the -# checkpoint; the card's quickstart names it explicitly). From -# peonist-ai/halogen-qwen3.8-flash-next @ 5cc17cea. -# The card does NOT establish this stock table as matched to the NATIVE -# conversion and no source-matched table is published, so the engine's own -# tools are the referee: NIAH 18/18 and PPL 3.968 on it (2026-10-17). It is -# HARDLINKED from the swift15 payload dir (a hardlink is a regular file; the -# serve gate refuses symlinks), i.e. byte-identical to the file that era -# served. +# The n-gram table is REQUIRED (the engine reads it beside the checkpoint) and +# disk-backed: mmap'd page cache, never pinned. It is a stock peonist companion +# hardlinked from the previous payload dir — a hardlink is a regular file, and +# the serve gate refuses symlinks — with the engine's own NIAH/PPL verdicts on +# the page as the referee. FRANK_DRAGOON_NGRAM_TABLE=qwen38-flash-next-ngram.hgn FRANK_DRAGOON_NGRAM_BYTES=51200246144 FRANK_DRAGOON_NGRAM_SHA256=3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe -# The vision tower: OPTIONAL on the engine, ENABLED here (image input). -# From peonist-ai/halogen-qwen3.8-flash-next @ e053f488. +# Vision tower: optional on the engine, ENABLED here (image input). FRANK_DRAGOON_VISION_TOWER=qwen38-flash-next-vision.hgn FRANK_DRAGOON_VISION_BYTES=897916416 FRANK_DRAGOON_VISION_SHA256=d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10 -# The tokenizer is a FLAT DIRECTORY, $FRANK_DRAGOON_MODEL_DIR/tokenizer, -# mounted whole at /tokenizer. The two repos ship the same one; fetched -# fresh from peonist @ e053f488 (tokenizer.json is LFS, byte-identical to -# the 27B-era copy; the rest are plain git blobs — size-checked only). The -# serve gate and the deployer refuse a missing file or ANY symlink: an HF -# cache snapshot symlinks each file into blobs/, and those links dangle +# The tokenizer is a FLAT DIRECTORY ($FRANK_DRAGOON_MODEL_DIR/tokenizer) mounted +# whole. The serve gate and the deployer refuse a missing file or ANY symlink: +# an HF cache snapshot symlinks each file into blobs/ and those links dangle # inside the container mount. FRANK_DRAGOON_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json" # --- engine flags (docs/FLAGS.md of the flash-server repo; anything unset # stays at the image default, which is what was qualified) ------------------ -# Served model id — Protoss name (wire rename 2026-10-09); the service, unit, -# scripts and this file keep the frank-dragoon name. OpenWebUI api_configs -# model_ids for the :7770 backend MUST name this value. +# Served id. OpenWebUI api_configs model_ids for the :7770 backend MUST name it. HALOGEN_MODEL_ID=frank-dragoon +# PER-REQUEST ceiling: a longer prompt is a 400 naming the limit, never a +# truncation. The harness window must sit ~45k UNDER this (the 400-zone law on +# the page) — a window equal to the ceiling makes every long request a 400. HALOGEN_CTX=200704 +# ONE pool behind both slots, sized so BOTH can carry a maximum-length request +# at once. Pool positions buy concurrency; the per-request window never moves +# with them. Legal band: ctx <= pool <= slots * ctx (the seam enforces it), and +# RAM the pool takes is RAM the n-gram table's page cache loses. HALOGEN_KV_POOL_POSITIONS=401408 +# Two concurrent conversations: the MTP drafter is not slot-locked on this +# engine, so the second slot is capacity, not a speculation tax. HALOGEN_KV_SLOTS=2 HALOGEN_HOST_RESERVE_GIB=20 -# Spec decode: MTP head is INSIDE the checkpoint (do NOT set HALOGEN_MTP_HEAD); -# depth 2 is the documented "best all-round" default. HALOGEN_DRAFTER / -# HALOGEN_DFLASH_* from the 27B engine DO NOT EXIST here. +# The MTP head is INSIDE the checkpoint — do NOT set HALOGEN_MTP_HEAD. +# 0 is mandatory: the adaptive default disarms the head for whole requests after +# low acceptance and halves decode. Depth 2 is the card's best-all-round default. HALOGEN_SPEC_ADAPT=0 -# Coupled pair (FLAGS.md table): the cap bounds how long ONE request holds -# the GPU, the timeout bounds how long the NEXT client waits. Do not move -# one alone (see the client-contract note above). +# Coupled trio: MAX_TOK is only the default for requests that omit max_tokens; +# the cap bounds how long ONE request holds the GPU and the queue timeout bounds +# how long the NEXT client waits. Move all three together or none. HALOGEN_MAX_TOK=8192 HALOGEN_MAX_TOKENS_CAP=65536 HALOGEN_QUEUE_TIMEOUT=7200 -# Server-side sampler = the card's thinking recipe (the engine now HAS one; -# requests may still override per-call). Matches models.yml extraBody. +# The engine has a server-side sampler (an absent temperature is NOT greedy +# here); models.yml extraBody carries the same recipe. HALOGEN_TEMPERATURE=1.0 HALOGEN_TOP_P=0.95 HALOGEN_TOP_K=20 -# Card's belt-and-braces: the engine pins weights by default -# (HALOGEN_FLASH_PIN_TRUNK=1; only the retired w4b build honored 0). +# Weights stay pinned: the engine pins by default and only the retired w4b build +# honored HALOGEN_FLASH_PIN_TRUNK=0, which this payload rejects. HALOGEN_WEIGHTS_LOCK=1 diff --git a/scripts/frank-zealot.env b/scripts/frank-zealot.env index aeead2d..48608ab 100644 --- a/scripts/frank-zealot.env +++ b/scripts/frank-zealot.env @@ -1,160 +1,56 @@ -# frank-zealot — llama.cpp (PrismML fork build 10777, CUDA) serving -# prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 **with the MTP head in the file** -# on the RTX 3090, :7771. +# frank-zealot — llama.cpp (PrismML fork build 10777, CUDA) on the RTX 3090, +# :7771. Payload: Ternary-Bonsai-2-9B PQ2_0 with the MTP head in the file. # -# 2026-10-18 (Frank): swap from the Bonsai-2-**27B** PQ2_0 to the 9B MTP export, -# and the same evening the shape moved to TWO 262,144-token conversations over an -# UNQUANTIZED (f16) KV cache — "shift frank-zealot to 2 parallels of unquantized -# kv". Exact K and V is the retrieval-fidelity ask; the price is one concurrent -# conversation, because f16 owns the card. Speculative decoding ships OFF — -# measured, not assumed; see the MTP law below, which is the whole story and the -# reason the publisher's "depth 2 works best" line does not transfer (and why -# exact KV makes the drafter illegal outright). -# -# THE RUNTIME IS A LOCK, NOT A PREFERENCE. The file is general.file_type 141 -# carrying custom GGML type 142 tensors plus prism.hadamard.* metadata -# (normalized-Sylvester-Walsh transform); stock llama.cpp refuses type 142 at -# load and a loader that ignored the Hadamard metadata would dequant WRONG. On -# top of that lock, MTP beside a quantized V cache needs fork build >= b10776, -# so the vendor prebuilt moved from 10754 (commit 2459f68b5, the 27B era) to -# 10777 (commit e3de71522). The 10754 tree stays on disk as the rollback -# runtime for the 27B payload. +# ONE OWNER: shape, payload identity, eras, laws and every measured number +# live in serverAppConfigs-docs/LLM/frank-zealot.md. This file carries the +# knobs and only the reason a reader editing THIS line cannot recover +# elsewhere. Where the two disagree, this file wins on values. LLAMA_BIN=/home/dank/GitPackages/llama-prism-b10777/llama-server -# 2,675,410,720 B, Hub LFS sha256 c79fcdfaf6b7… (verified at fetch by -# scripts/download-frank-zealot-bonsai9.sh). This IS the MTP build: the draft -# head ships INSIDE the checkpoint, so MTP availability is a property of the -# FILE, never of a flag. The plain PQ2_0 sibling could not run it at any flag -# value; loading this file instead of that one is the only change that matters. +# The fork is a LOCK, not a preference: custom tensor type 142 + prism.hadamard.* +# metadata. Stock llama.cpp refuses the file; a loader ignoring the Hadamard +# metadata dequantizes wrong, silently. MODEL_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf -# 624,229,664 B, Hub LFS sha256 e0cb6f69… — Q8_0 IS MANDATORY, not a preference: -# the publisher's card says the BF16 mmproj "can crash llama-server when several -# image requests run at once; Q8_0 is not affected", and this endpoint runs -# --parallel 2, i.e. exactly that shape. The projector's worst case lands AFTER -# the engine's own fit check, so it is never inside a projection number. +# Q8_0 IS MANDATORY: the publisher states the BF16 tower can crash llama-server +# under concurrent image requests, which is exactly --parallel 2. MMPROJ_FILE=/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf -# Base alignment: Bonsai-2 is the censored BASE lineage. The 27B CRACK re-export -# (/home/dank/models/Bonsai2-CRACK/) is byte-identical geometry to the 27B it -# matched, NOT to this 9B file — if a refusal ever surfaces there is no -# one-line swap here; the 9B dealigned export would have to be fetched first. -# -# Served model id — the harness/family name. The SERVICE, unit, scripts and -# this env file keep the frank-zealot name; only the alias on the wire is the -# model id. Consumers that key on the served id: OpenWebUI (model row, -# access_grant, api_configs model_ids) and the harness models.yml. +# Served id — the harness/family name, not a fact about the payload. +# Id-keyed consumers: OpenWebUI (model row, access_grant, api_configs), models.yml. ALIAS=frank-zealot BIND_ADDR=0.0.0.0 LISTEN_PORT=7771 -# CTX_SIZE is the POOL llama.cpp divides across slots: 524,288 / 2 = 262,144 per -# conversation — each slot STILL at the model's TRAINED window; the per-slot -# contract did not move. MUST stay divisible by PARALLEL (the seam enforces it) -# and the harness contextWindow quotes the PER-SLOT number — a longer prompt is -# answered with a 400, never truncated. THE POOL CANNOT STAY AT 786,432 UNDER -# EXACT KV: f16 bills 32,768 B/token, so that pool is 24,576 MiB — more than the -# 24,123 MiB the driver leaves addressable, before a single weight loads. -# Halving the POOL to 524,288 (16,384 MiB) is what funds exact KV. The window -# cannot go past 262,144 a slot either: that is the native wall (no YaRN in this -# path), so a 3rd slot would be a KV-dtype cut, not more context. +# CTX_SIZE is the POOL llama.cpp divides across slots: 524,288 / 2 = 262,144 a +# conversation, each still at the trained window. MUST stay divisible by +# PARALLEL (the seam enforces it) and models.yml contextWindow quotes the +# PER-SLOT number — a longer prompt is a 400, never a truncation. The pool +# cannot go back to 786,432 under exact f16 KV: 24,576 MiB of cache against the +# 24,123 MiB the driver leaves addressable, before a weight loads. CTX_SIZE=524288 PARALLEL=2 BATCH_SIZE=4096 -# Ubatch is a prefill knob (decode never moves) with NO interior optimum on this -# CUDA card up to 2048 — 2048 is the measured optimum (+4.6 % prefill). It has a -# second job here: it sets the compute-buffer size, and the compute buffers are -# what decide whether MTP fits (see the MTP law). 1024 is the documented MTP arm. +# Ubatch is a prefill knob (decode never moves with it) and it sets the +# compute-buffer size — which is what decides whether a drafter fits. +# UBATCH_SIZE=1024 is the documented MTP arm. UBATCH_SIZE=2048 -# KV law (header + boot ledger, both confirmed): qwen35 hybrid, 32 layers with -# qwen35.full_attention_interval = 4 ⇒ 8 full-attention + 24 Gated DeltaNet; -# 4 KV heads × 256 dim (n_embd_k_gqa = n_embd_v_gqa = 1,024 per layer) = 8,192 -# elements per token per cache. EXACT KV: f16 = 2 B/element ⇒ 16,384 B per cache -# ⇒ 32,768 B/token = 32 KiB, so the 524,288 pool is 16,384.00 MiB. q8_0 bills -# 1.0625 B/element (32 codes + one 2-byte scale per 32-block) ⇒ 17,408 B/token, -# which is the 13,056.00 MiB the 3-slot ledger printed at the 786,432 pool. -# THE RATIO IS 1.882×, NOT 2×: the old note here said "f16 would bill 26,112 -# MiB" because it doubled the q8_0 figure with its scale overhead included; -# exact arithmetic says 24,576 MiB at that pool. The wall stands, only its -# height moves. The 24 DeltaNet layers bill a fixed recurrent state per SLOT -# instead of per token — the 3-slot ledger printed "RS buffer size = 150.75 MiB -# (3 cells, 32 layers, 3 seqs)", ~50.25 MiB a cell so two slots bill 100.50 — -# while the SPEC-ARMED arms print 452.25 MiB, because MTP depth 2 bills three -# seqs per slot (9 cells). Pool arithmetic alone under-prices the shape. +# Exact K and V: the retrieval-fidelity ask, paid with one conversation of +# concurrency. Rate law and the arithmetic: [[frank-zealot]] → Laws. CACHE_TYPE_K=f16 CACHE_TYPE_V=f16 -# SPECULATIVE DECODING: OFF, MEASURED. The head is in the file and the drafter -# is GOOD (acceptance 0.798/0.617 per position, mean accepted length 2.41) — it -# still loses, because the draft context bills its OWN f16 KV over the WHOLE -# pool at its default dtype. That dtype IS a knob (--spec-draft-type-k / -# --spec-draft-type-v, both default f16 — verified against this build's --help), -# but SPEC_TYPE=none never sends them. Under the -# 3-slot q8_0 shape that bill was 4,096 B/position × 786,432 = 3,072.00 MiB plus -# ~1,758 MiB of buffers. Booted arms, same card, same traffic (2026-10-18, the -# WHOLE table is q8_0 K + q8_0 V — it predates exact KV): -# 3 × 262,144 spec OFF ub 2048 → 17,876 used / 6,249 free, decode 167.2 t/s -# 3 × 262,144 spec ON ub 2048 → 22,706 used / 1,419 free, decode 109.2 t/s -# 3 × 262,144 spec ON ub 1024 → 21,400 used / 2,725 free, decode 213.5 t/s -# 2 × 262,144 spec ON ub 2048 → 17,178 used / 6,947 free, decode 213.2 t/s -# 1 × 262,144 spec ON ub 2048 → 11,652 used / 12,473 free, decode 214.2 t/s -# MTP is worth +28 % decode whenever the card has room. At the SHIPPED exact-KV -# shape an f16-draft arm has none: the draft KV alone is 4,096 × 524,288 = -# 2,048.00 MiB plus its buffers, on a trunk that already projects ~21,200 MiB of -# the 24,123 the driver leaves addressable. DOORS BACK TO THE DRAFTER: one -# window per conversation; KV back to q8_0 with UBATCH_SIZE=1024 (213.5 t/s); -# or the UNMEASURED arm --spec-draft-type-k q8_0 --spec-draft-type-v q8_0, which -# halves the draft bill to 1,088 MiB and is legal on this build (>= b10776). -# Nobody has booted that beside exact KV — measure before promising it. -# This endpoint's load is still the fleet's COMPACTION HUB (prefill-bound, -# ~170k in): prefill is weight-bandwidth bound and does not read the KV cache -# per generated token, which is why exact KV is affordable here at all; what it -# buys back is concurrency — TWO conversations, not three. -# SPEC_TYPE=none strips the WHOLE spec set (--spec-type, every --spec-draft-*): -# flags move as a set or the arm measures a crash instead of an absence. +# Speculation OFF — measured, not assumed: the draft context bills its own f16 +# KV over the WHOLE pool, and there is no room beside exact KV (the arm census +# is on the page). SPEC_TYPE=none strips the WHOLE set (--spec-type and every +# --spec-draft-*); half a set has segfaulted at load. SPEC_TYPE=none SPEC_DRAFT_N_MAX=2 SPEC_DRAFT_N_MIN=0 SPEC_DRAFT_P_MIN=0 -# The card's sampler recipe lives in the checkpoint (general.sampling.* = temp -# 1.0 / top_p 0.95 / top_k 20 / min_p 0 / repeat_penalty 1.0 — confirmed in the -# engine's own /props defaults) and llama.cpp reads it itself. Presence penalty -# is the ONE recipe value with no GGUF metadata key, so it must reach the argv. -# PRESENCE_PENALTY="" is the documented opt-out. +# The sampler recipe lives in the checkpoint (general.sampling.*) and +# llama.cpp reads it itself. Presence penalty is the one recipe value with no +# GGUF metadata key, so it must reach argv. "" is the documented opt-out. PRESENCE_PENALTY=1.0 NGL=99 -# Boot ledger REFERENCE — the previous production boot (2026-10-18 13:19, -# 3 × 262,144, q8_0 K + q8_0 V, verbosity 4), i.e. the shape this file replaced: -# 17,956 MiB used / 6,169 MiB free of 24,123 addressable (nvidia-smi settled; -# its total column reads 24,576 — the driver reserves the difference, and -# older notes quoting "a 24,123 MiB card" mean this addressable figure) -# CUDA0 model 2,036.71 MiB + CPU_Mapped 257.66 MiB -# CUDA0 KV 13,056.00 MiB (q8_0 K + q8_0 V, 786,432 positions) -# CUDA0 RS 150.75 MiB (3 cells, 32 layers, 3 seqs) -# compute 1,409.02 MiB CUDA + 1,088.87 MiB CUDA_Host (+2.84 MiB output buf) -# mmproj 595 MiB weights + 248.10 MiB CUDA compute meta; estimated worst case -# 868.32 MiB — charged AFTER the engine's own fit check, as always -# no clamp line, no cudaMalloc error -# Measured on THAT boot (/completion timings, greedy, ignore_eos, 512 tokens): -# 168.7 t/s decode (three runs 168.4-168.7), ~300 t/s aggregate across THREE -# concurrent streams (102.8 each), 2,877 t/s prefill at 106,418. The 27B it -# replaced gave 76.6 / 109.4 / 691.7 at 166k. These are the numbers the exact-KV -# arm is judged against: decode reads 1.882× the KV bytes per token, so a decode -# drop is EXPECTED and must be measured, not assumed, while prefill should hold. -# Every non-KV term of that ledger survives this reslice except RS (−50.25 MiB, -# one fewer cell) and one slot's worth of scheduler state, so the f16 arm -# PROJECTS 17,956 − 13,056 + 16,384 − 50 = 21,234 MiB used / 2,889 free, and -# ~2.0 GiB free through the projector's worst case. REPLACE THIS PROJECTION WITH -# THE MEASURED PAIR after the boot: read the new ledger AND settled nvidia-smi. -# Fallback rungs if that margin gets eaten (desktop, co-tenant), cheapest first: -# CACHE_TYPE_V=q8_0 (exact K, quantized V) frees 8,192 MiB at this pool; -# CTX_SIZE=401408 (two 200,704 conversations) puts exact KV at 12,544.00 MiB and -# buys ~6.7 GiB back; UBATCH_SIZE=1024 hands 1,306 MiB of compute buffers back. -# Log verbosity threshold (llama.cpp -lv). 4 = debug: KV layout, buffer sizes -# and graph capture land in the journal, which is what the VRAM-fit forensics -# read. Levels above 4 flood the journal. +# 4 = debug: KV layout, buffer sizes and graph capture land in the journal, and +# that is what VRAM forensics read. Above 4 floods the journal. VERBOSITY=4 - -# Serve /metrics. llama.cpp answers 501 without --metrics, and Dankclaw's -# Prometheus scrapes /metrics on every fleet endpoint — a 501 reads as DOWN, so -# this service has held the critical LlmEndpointDown alert for as long as the -# metric has history (verified 2026-10-18: the oldest retained sample of -# up{job="frank-zealot"}, 2026-10-11, is already 0; scrapeUrl -# http://host.containers.internal:7771/metrics, lastError "server returned HTTP -# status 501 Not Implemented"). 0 opts out and re-breaks the scrape. +# Without --metrics llama.cpp answers 501 on /metrics, which Prometheus reads +# as DOWN while /health stays 200. 0 opts out and re-breaks the scrape. METRICS=1