Frank
2b127163c8
frank-dragoon's total in-flight context was HALF frank-zealot's: zealot runs one 401,408-position pool split into two 200,704 conversations, while dragoon ran ONE 200,704-position pool behind its two slots, so a single maximum-length request filled it and two long conversations had to stay under ~100k each. HALOGEN_KV_POOL_POSITIONS 200704 -> 401408 = slots x ctx, the top of the seam's legal band (ctx <= pool <= slots * ctx). The PER-REQUEST ceiling does not move: HALOGEN_CTX stays 200704, because the checkpoint's native window is 262,144 and the engine's own 400 text refuses to exceed it without HALOGEN_ROPE_YARN. What this buys is concurrency, not a longer conversation, which is why the harness contextWindow (155648 = ceiling - 45056) is untouched. Cost: ~10.8 GiB of KV where 200,704 cost 5.5 (262,144 cost 7.2; the 2026-10-06 shape's 524,288 measured 14.6), taken from the page cache the 51.2 GB n-gram table reads through — the warm prompt-cache ratio is the number to re-measure after the boot. Engine-side total goes 74.1 -> ~79.4 GiB of 121.2. Also carries scripts/download-frank-zealot-bonsai2-ptq1.sh: the PTQ1_0 candidate payload fetcher (5,946,648,928 B, LFS sha256-verified, mmproj hardlinked), with the publisher's own throughput table in its header — the packing is smaller but the card measures it SLOWER on Ampere, so the swap is decided on our card, not on the card's A100 row.
207 lines
13 KiB
Bash
207 lines
13 KiB
Bash
# frank-dragoon.env — tunables for the frank-dragoon endpoint on Dankclaw's
|
||
# Strix Halo iGPU (gfx1151), host port 7770.
|
||
#
|
||
# RUNTIME (2026-10-17, Frank): halogen-flash-server 0.16.2 — the
|
||
# closed-source C++/HIP engine (image ghcr.io/peonist-ai/halogen-flash-server,
|
||
# digest-pinned below) serving Quat3rnion/halogen-qwen3.8-flash-next-v2-
|
||
# abliterated — the NATIVE v2 conversion (2026-10-11) of orcarouter's pinned
|
||
# BF16 Qwen3.8-Flash-Next, abliterated — as a podman container under a root
|
||
# unit, driven by scripts/serve-frank-dragoon.sh. This CUTS OVER the
|
||
# 2026-10-06 Swift1.5-finetune shape; rollback = revert the swap commit and
|
||
# redeploy: the swift15 payload stays on disk at
|
||
# /home/dank/models/halogen-flashnext-v2/, the 27B-era payload beside it.
|
||
#
|
||
# MEASURED A/B, both boots on this box, identical battery traffic (engine
|
||
# tools with the service stopped, behavior probes against it, 2026-10-17):
|
||
# NIAH 18/18 vs 17/18, refusal census 0/5 on both, tool calls and the
|
||
# red-PNG vision probe identical, needle hit at both depths of a 21.5k-token
|
||
# haystack, decode 50.0 vs 46.8 t/s, cold prefill 1,604 vs 1,425 t/s,
|
||
# wikitext-2 PPL 3.968 vs 3.877 (the finetune's edge is gone: +2.3 %) and
|
||
# thinking tokens 1,476 vs 784 mean — the finetune compressed reasoning, the
|
||
# base thinks ~1.9x longer per answer.
|
||
#
|
||
# The A/B ran at the card's ctx/pool of 262,144. Afterwards Frank lowered ctx to
|
||
# 200,704 (RAM + the harness's compaction budget) and, on 2026-10-18, raised the
|
||
# pool to 2 x ctx = 401,408 so BOTH slots can carry a full-length request at
|
||
# once; the rates above are the ceiling-262,144 record and the production ledger
|
||
# is the ctx-200,704 one.
|
||
#
|
||
# WHY THE RUNTIME MOVED WITH THE MODEL: the payload is .hgn — per its own
|
||
# card it "will not load in transformers, vLLM, or llama.cpp" — and the v2
|
||
# .hgn loads ONLY in the flash-server line (the old halogen engine is for
|
||
# the 27B p1w4d file; the v2 quickstart pins flash-server). What the move
|
||
# buys (card + live probe evidence): MTP spec decode (53-57 t/s greedy,
|
||
# 3-10x serial), ~1,640 t/s prefill at 34k, ~72% fewer thinking tokens than
|
||
# the censored 27B (905 mean), the ABLITERATED base (refusals should NOT
|
||
# return), IMAGE INPUT (the v2 vision tower loads beside the checkpoint —
|
||
# red-PNG probe verified on the card; the 27B engine ignored its encoder),
|
||
# tool calls verified, and the native 262,144 context — served at 200,704 by
|
||
# choice from 2026-10-17 (see SHAPE: the ceiling is a RAM decision, not a
|
||
# capability limit).
|
||
#
|
||
# SHAPE — the knobs that are NOT free, with the arithmetic:
|
||
# HALOGEN_CTX=200704 is the PER-REQUEST ceiling and the per-conversation
|
||
# hard limit — a prompt over it is a 400 naming the limit, never a silent
|
||
# truncation. LOWERED from 262,144 on 2026-10-17 at Frank's direction (the
|
||
# checkpoint's native window is 262,144 and the engine serves anything
|
||
# below it): every position costs KV pool RAM and shrinks the prompt-cache
|
||
# entry (111 MiB at 262,144, ~85 MiB here). The 400-zone law is INVARIANT
|
||
# to the value — it says the harness window must sit ~45k UNDER the engine
|
||
# ceiling, because the harness clamps max_tokens to contextWindow − est −
|
||
# 4096, its chars/4 est UNDER-counts the rendered prompt by ~3 % on
|
||
# code/agent traffic, and compaction only fires at 85 % (measured 2026-10-07
|
||
# at the old 200,192 ceiling: est 152,938, server 157,388, over by 354).
|
||
# Harness contract (models.yml): contextWindow 155648 (= ceiling − 45056,
|
||
# the gap that absorbs the estimate error), maxTokens 45056.
|
||
# Compaction fires at 85 % = 132,301 est ≈ 136k measured — the 180k target
|
||
# of the 262,144 era is unreachable at this ceiling: the window itself is
|
||
# 155,648. Worst-case request: prompt + budget <= 155648 − 4096 + 3 %
|
||
# est-error ≈ 156.2k <= 200,704 — ~9.5× the 3 % error of margin (robust to a
|
||
# ~25 % under-count). Compaction rides on frank-zealot: input ≈ threshold −
|
||
# keepRecent(20k) ≈ 112k, inside its 200,704 slot.
|
||
# HALOGEN_KV_POOL_POSITIONS=401408 = SLOTS x CTX (2 x 200,704), the top of the
|
||
# legal band (the seam enforces ctx <= pool <= slots * ctx). Still ONE pool
|
||
# behind HALOGEN_KV_SLOTS=2 (ledger form: "2 slots over ONE <pool>-position KV
|
||
# pool, each request up to HALOGEN_CTX"), but sized so BOTH slots can hold a
|
||
# MAXIMUM-length request at the same time — the parity with frank-zealot's
|
||
# 401,408-position pool that Frank ordered on 2026-10-18. The PER-REQUEST
|
||
# ceiling does NOT move with it: HALOGEN_CTX stays 200,704, because the
|
||
# checkpoint's native window is 262,144 and the engine refuses to go past it
|
||
# without HALOGEN_ROPE_YARN (its own 400 text says so). ~28 KiB/position
|
||
# (FLAGS.md) ⇒ ~10.8 GiB, where pool == ctx (200,704) cost 5.5, 262,144 cost
|
||
# 7.2 and the 2026-10-06 shape's 524,288 measured 14.6. The engine's preflight
|
||
# formula bills a little more than the reservation (block scratch + ~115
|
||
# MiB/slot of O(1) state) — quote the RESERVATION. FLAGS.md law: "RAM the pool
|
||
# takes is RAM that table loses" — the 47.68 GiB n-gram table surrenders this
|
||
# much page cache, so the warm prompt-cache ratio is what to re-measure.
|
||
# HALOGEN_KV_SLOTS=2 restores the w4b-era shape on this box: the MTP
|
||
# drafter is NOT slot-locked on this engine (batching and speculation
|
||
# coexist — DFlash-era slots>1 law does NOT carry over; the card's own
|
||
# quickstart ships 2 slots WITH the head), so the second slot is
|
||
# concurrent capacity, not a speculation tax.
|
||
# Memory ledger against the box's 121.2 GiB unified (the n-gram table is
|
||
# mmap'd page cache beside the checkpoint, not a pinned charge — the boot
|
||
# ledger agrees: "mapped 51.2 GB, NOT pinned (Pin::None)"):
|
||
# weights ~62.9 GiB locked (0.8 of them the vision tower) + KV ~10.8 +
|
||
# working ~5.7 (HALOGEN_MAX_TOK=8192) ≈ 79.4 GiB engine-side, with the
|
||
# prompt cache (~1.0 at this ctx; 1.3 was measured at 262144) booked
|
||
# separately by the ledger; HALOGEN_HOST_RESERVE_GIB=20 keeps the host
|
||
# honest. Against the 121.2 GiB box that leaves ~22 GiB over the reserve for
|
||
# baseline co-tenancy (frank-zealot, ComfyUI, Caddy/observ ~10-15). The boot
|
||
# ledger is the referee — read it after deploy and re-size if the engine
|
||
# prints a clamp.
|
||
# HALOGEN_SPEC_ADAPT=0 is MANDATORY (card): the default adaptive policy
|
||
# turns the MTP head off for whole requests after low acceptance and
|
||
# HALVES decode as a result. Frank accepted the head always-on trade
|
||
# (temp 1.0 traffic costs acceptance; the card still ships the recipe).
|
||
#
|
||
# CLIENT CONTRACT deltas vs the 27B shape (harness side lives in
|
||
# ~/.omp/agent/models.yml):
|
||
# * SERVER-SIDE SAMPLER EXISTS NOW: HALOGEN_TEMPERATURE/TOP_P/TOP_K set
|
||
# the card's thinking recipe (1.0 / 0.95 / 20) as engine defaults, so
|
||
# an absent temperature is no longer greedy. models.yml extraBody
|
||
# carries the same values — consistent, not load-bearing.
|
||
# * HALOGEN_MAX_TOK=8192 is only the DEFAULT for requests that omit
|
||
# max_tokens; the harness sends at most its maxTokens 45056. Requests
|
||
# over HALOGEN_MAX_TOKENS_CAP (65536) are a 400 naming the cap. Cap and
|
||
# queue timeout are COUPLED (FLAGS.md table): 65536 at ~10 t/s serial
|
||
# worst case needs > 6550 s; 7200 honors it. Do not move one alone.
|
||
# * max_tokens INCLUDES thinking; a reasoning reply over a small budget
|
||
# returns empty content with finish_reason "length".
|
||
# * Prompt cache is ON by default (PROMPT_CACHE=2): a follow-up turn at
|
||
# 100k context re-prefills ~2 s instead of ~88 s cold. No pin needed —
|
||
# unlike the 27B engine, auto-sizing fits this box's ledger.
|
||
# * Thinking lands in reasoning_content (same as the 27B engine); the
|
||
# v2 chat template rejects reasoning_effort outside xhigh|medium|low,
|
||
# so the models.yml fold (high->xhigh) still applies.
|
||
# * Vision: OpenAI image_url content parts only (the fleet's standard
|
||
# contract). >=256 px probe images or the model answers "video".
|
||
# * Concurrency at depth is tight now: the KV pool is ONE 200,704-position
|
||
# budget behind 2 slots — exactly one maximum-length request fills it. Two
|
||
# conversations coexist only while they stay under ~100k each. The
|
||
# PER-REQUEST ceiling moved WITH the pool (262144 -> 200704), so the harness
|
||
# contract moves too: contextWindow 155648 = ceiling − 45k, never higher.
|
||
# * response_format json_object/json_schema unsupported in this line
|
||
# (no constrained decoding) — unchanged from the 27B engine.
|
||
# * The engine's token protocol has no auth and binds loopback INSIDE
|
||
# the container; only the frontend port (8731) is published. Same as
|
||
# before. Kernel needs CONFIG_HSA_AMD_SVM (CachyOS ships it) and
|
||
# amdgpu.noretry=0 (check at deploy).
|
||
|
||
# Image: digest-pinned (fetch law: tag pulls silently re-point; the digest
|
||
# is what was qualified). 0.16.2 = the version the model card names as its
|
||
# quickstart pin; RepoDigest captured at pull on 2026-10-06.
|
||
FRANK_DRAGOON_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
|
||
# Container identity: kept from the 27B era (unit ExecStop names it).
|
||
FRANK_DRAGOON_CONTAINER=frank-dragoon-halogen
|
||
# Old quadlet-era container identity; the serve script clears stale names.
|
||
FRANK_DRAGOON_LEGACY_CONTAINERS=frank-dragoon
|
||
# host:container — the frontend listens on 8731 inside (HALOGEN_API_PORT
|
||
# default); the engine port (8730) is never published.
|
||
FRANK_DRAGOON_PUBLISH=7770:8731
|
||
|
||
FRANK_DRAGOON_MODEL_DIR=/home/dank/models/halogen-v2-abliterated
|
||
# The checkpoint: Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated
|
||
# qwen38-flash-next-v2-abliterated.hgn — the NATIVE 2026-10-11 v2 conversion
|
||
# of orcarouter's pinned BF16 Qwen3.8-Flash-Next, abliterated. NOT the writer
|
||
# transplant this filename used to carry: 66,687,678,432 B where the swift15
|
||
# r2 measured 66,687,678,464 B — same filename family, different artifact.
|
||
# Hub LFS sha256 verified at fetch by scripts/download-frank-dragoon-v2abl.sh.
|
||
FRANK_DRAGOON_CHECKPOINT=qwen38-flash-next-v2-abliterated.hgn
|
||
FRANK_DRAGOON_CHECKPOINT_BYTES=66687678432
|
||
FRANK_DRAGOON_CHECKPOINT_SHA256=b5bd925fa2b44343be4a2cd328cae6b57ce7ffa2fe2cc56281717377e5a1ce8a
|
||
# The n-gram lookup table: REQUIRED (the engine reads it beside the
|
||
# checkpoint; the card's quickstart names it explicitly). From
|
||
# peonist-ai/halogen-qwen3.8-flash-next @ 5cc17cea.
|
||
# The card does NOT establish this stock table as matched to the NATIVE
|
||
# conversion and no source-matched table is published, so the engine's own
|
||
# tools are the referee: NIAH 18/18 and PPL 3.968 on it (2026-10-17). It is
|
||
# HARDLINKED from the swift15 payload dir (a hardlink is a regular file; the
|
||
# serve gate refuses symlinks), i.e. byte-identical to the file that era
|
||
# served.
|
||
FRANK_DRAGOON_NGRAM_TABLE=qwen38-flash-next-ngram.hgn
|
||
FRANK_DRAGOON_NGRAM_BYTES=51200246144
|
||
FRANK_DRAGOON_NGRAM_SHA256=3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe
|
||
# The vision tower: OPTIONAL on the engine, ENABLED here (image input).
|
||
# From peonist-ai/halogen-qwen3.8-flash-next @ e053f488.
|
||
FRANK_DRAGOON_VISION_TOWER=qwen38-flash-next-vision.hgn
|
||
FRANK_DRAGOON_VISION_BYTES=897916416
|
||
FRANK_DRAGOON_VISION_SHA256=d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10
|
||
# The tokenizer is a FLAT DIRECTORY, $FRANK_DRAGOON_MODEL_DIR/tokenizer,
|
||
# mounted whole at /tokenizer. The two repos ship the same one; fetched
|
||
# fresh from peonist @ e053f488 (tokenizer.json is LFS, byte-identical to
|
||
# the 27B-era copy; the rest are plain git blobs — size-checked only). The
|
||
# serve gate and the deployer refuse a missing file or ANY symlink: an HF
|
||
# cache snapshot symlinks each file into blobs/, and those links dangle
|
||
# inside the container mount.
|
||
FRANK_DRAGOON_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json"
|
||
|
||
# --- engine flags (docs/FLAGS.md of the flash-server repo; anything unset
|
||
# stays at the image default, which is what was qualified) ------------------
|
||
# Served model id — Protoss name (wire rename 2026-10-09); the service, unit,
|
||
# scripts and this file keep the frank-dragoon name. OpenWebUI api_configs
|
||
# model_ids for the :7770 backend MUST name this value.
|
||
HALOGEN_MODEL_ID=frank-dragoon
|
||
HALOGEN_CTX=200704
|
||
HALOGEN_KV_POOL_POSITIONS=401408
|
||
HALOGEN_KV_SLOTS=2
|
||
HALOGEN_HOST_RESERVE_GIB=20
|
||
# Spec decode: MTP head is INSIDE the checkpoint (do NOT set HALOGEN_MTP_HEAD);
|
||
# depth 2 is the documented "best all-round" default. HALOGEN_DRAFTER /
|
||
# HALOGEN_DFLASH_* from the 27B engine DO NOT EXIST here.
|
||
HALOGEN_SPEC_ADAPT=0
|
||
# Coupled pair (FLAGS.md table): the cap bounds how long ONE request holds
|
||
# the GPU, the timeout bounds how long the NEXT client waits. Do not move
|
||
# one alone (see the client-contract note above).
|
||
HALOGEN_MAX_TOK=8192
|
||
HALOGEN_MAX_TOKENS_CAP=65536
|
||
HALOGEN_QUEUE_TIMEOUT=7200
|
||
# Server-side sampler = the card's thinking recipe (the engine now HAS one;
|
||
# requests may still override per-call). Matches models.yml extraBody.
|
||
HALOGEN_TEMPERATURE=1.0
|
||
HALOGEN_TOP_P=0.95
|
||
HALOGEN_TOP_K=20
|
||
# Card's belt-and-braces: the engine pins weights by default
|
||
# (HALOGEN_FLASH_PIN_TRUNK=1; only the retired w4b build honored 0).
|
||
HALOGEN_WEIGHTS_LOCK=1
|