Frank
31b1ec397b
Engine: halogen-flash-server 0.16.2, digest-pinned (pulled 0c61bf84). Payload: Quat3rnion qwen38-flash-next-v2-swift15-abliterated.hgn (66,687,678,464 B, sha256-verified) + REQUIRED n-gram table (51,200,246,144 B @peonist 5cc17cea) + vision tower (897,916,416 B @peonist e053f488) + shared tokenizer (6 files). Shape: 2 slots x 200192 (KV pool 400384 positions, ~10.7 GiB), MTP always-on (HALOGEN_SPEC_ADAPT=0, card-mandated), server-side sampler at the card's 1.0/0.95/20, MAX_TOK 8192 with the 65536/7200 coupled pair, vision ON. Rollback = revert this commit (27B payload stays on disk). - serve script: byte-exact gates on all three .hgn files + flat tokenizer, foreground podman run (ledger to the log file) - deploy-remote: twin gates for n-gram/vision, flashnext fetch pointer - seam: 19 tests rewritten (spec-adapt law, pool product, compaction math, render shape); full suite 362/362 green
146 lines
8.7 KiB
Bash
146 lines
8.7 KiB
Bash
# frank-large.env — tunables for the frank-large endpoint on Dankclaw's
|
|
# Strix Halo iGPU (gfx1151), host port 7770.
|
|
#
|
|
# RUNTIME (2026-10-06, Frank): halogen-flash-server 0.16.2 — the
|
|
# closed-source C++/HIP engine (image ghcr.io/peonist-ai/halogen-flash-server,
|
|
# digest-pinned below) serving the Quat3rnion ABLITERATED Swift1.5 finetune
|
|
# of peonist's Qwen3.8-Flash-Next v2 checkpoint, as a podman container under
|
|
# a root unit, driven by scripts/serve-frank-large.sh. This CUTS OVER the
|
|
# 2026-10-05 halogen 27B-dense shape (censored base, DFlash2, 31.7 t/s):
|
|
# rollback = revert the swap commit's trio; the 27B payload stays on disk at
|
|
# /home/dank/models/halogen-qwen3.8-27b/.
|
|
#
|
|
# WHY THE RUNTIME MOVED WITH THE MODEL: the payload is .hgn — per its own
|
|
# card it "will not load in transformers, vLLM, or llama.cpp" — and the v2
|
|
# .hgn loads ONLY in the flash-server line (the old halogen engine is for
|
|
# the 27B p1w4d file; the v2 quickstart pins flash-server). What the move
|
|
# buys (card + live probe evidence): MTP spec decode (53-57 t/s greedy,
|
|
# 3-10x serial), ~1,640 t/s prefill at 34k, ~72% fewer thinking tokens than
|
|
# the censored 27B (905 mean), the ABLITERATED base (refusals should NOT
|
|
# return), IMAGE INPUT (the v2 vision tower loads beside the checkpoint —
|
|
# red-PNG probe verified on the card; the 27B engine ignored its encoder),
|
|
# tool calls verified, and the native 262,144 context.
|
|
#
|
|
# SHAPE — the knobs that are NOT free, with the arithmetic:
|
|
# HALOGEN_CTX=200192 is the PER-SLOT window and the per-conversation hard
|
|
# ceiling — a prompt over it is a 400 naming the limit, never a silent
|
|
# truncation. Harness compaction fires at 70% = 140,134, and 140,134 +
|
|
# 57,344 (harness maxTokens) = 197,478 <= 200,192 keeps prompt + budget
|
|
# inside the slot. models.yml contextWindow must quote 200192 or the
|
|
# harness discovers the wall as a 400.
|
|
# HALOGEN_KV_POOL_POSITIONS=400384 = KV_SLOTS(2) x CTX — the explicit
|
|
# form of the engine's default, so the seam test can pin the product.
|
|
# KV costs ~28 KiB/position (FLAGS.md) => 400,384 x 28 KiB = 10.7 GiB.
|
|
# HALOGEN_KV_SLOTS=2 restores the w4b-era shape on this box: the MTP
|
|
# drafter is NOT slot-locked on this engine (batching and speculation
|
|
# coexist — DFlash-era slots>1 law does NOT carry over; the card's own
|
|
# quickstart ships 2 slots WITH the head), so the second slot is
|
|
# concurrent capacity, not a speculation tax.
|
|
# Memory ledger against the box's 121.2 GiB unified (the n-gram table is
|
|
# mmap'd page cache beside the checkpoint, not a pinned charge — the
|
|
# card's own quickstart arithmetic does not bill it):
|
|
# weights ~62 GiB resident + KV 10.7 + arena (HALOGEN_MAX_TOK=8192 =>
|
|
# ~8 GiB per FLAGS.md arena formula 3.8+8.4*(MT/16384)) + vision ~2
|
|
# = ~82.7 GiB engine-side; HALOGEN_HOST_RESERVE_GIB=20 keeps the host
|
|
# honest; baseline co-tenancy (frank-small, ComfyUI, Caddy/observ)
|
|
# ~10-15. ~18 GiB margin. The boot ledger is the referee — read it
|
|
# after deploy and re-size if the engine prints a clamp.
|
|
# HALOGEN_SPEC_ADAPT=0 is MANDATORY (card): the default adaptive policy
|
|
# turns the MTP head off for whole requests after low acceptance and
|
|
# HALVES decode as a result. Frank accepted the head always-on trade
|
|
# (temp 1.0 traffic costs acceptance; the card still ships the recipe).
|
|
#
|
|
# CLIENT CONTRACT deltas vs the 27B shape (harness side lives in
|
|
# ~/.omp/agent/models.yml):
|
|
# * SERVER-SIDE SAMPLER EXISTS NOW: HALOGEN_TEMPERATURE/TOP_P/TOP_K set
|
|
# the card's thinking recipe (1.0 / 0.95 / 20) as engine defaults, so
|
|
# an absent temperature is no longer greedy. models.yml extraBody
|
|
# carries the same values — consistent, not load-bearing.
|
|
# * HALOGEN_MAX_TOK=8192 is only the DEFAULT for requests that omit
|
|
# max_tokens; the harness always sends 57344. Requests over
|
|
# HALOGEN_MAX_TOKENS_CAP (65536) are a 400 naming the cap. Cap and
|
|
# queue timeout are COUPLED (FLAGS.md table): 65536 at ~10 t/s serial
|
|
# worst case needs > 6550 s; 7200 honors it. Do not move one alone.
|
|
# * max_tokens INCLUDES thinking; a reasoning reply over a small budget
|
|
# returns empty content with finish_reason "length".
|
|
# * Prompt cache is ON by default (PROMPT_CACHE=2): a follow-up turn at
|
|
# 100k context re-prefills ~2 s instead of ~88 s cold. No pin needed —
|
|
# unlike the 27B engine, auto-sizing fits this box's ledger.
|
|
# * Thinking lands in reasoning_content (same as the 27B engine); the
|
|
# v2 chat template rejects reasoning_effort outside xhigh|medium|low,
|
|
# so the models.yml fold (high->xhigh) still applies.
|
|
# * Vision: OpenAI image_url content parts only (the fleet's standard
|
|
# contract). >=256 px probe images or the model answers "video".
|
|
# * response_format json_object/json_schema unsupported in this line
|
|
# (no constrained decoding) — unchanged from the 27B engine.
|
|
# * The engine's token protocol has no auth and binds loopback INSIDE
|
|
# the container; only the frontend port (8731) is published. Same as
|
|
# before. Kernel needs CONFIG_HSA_AMD_SVM (CachyOS ships it) and
|
|
# amdgpu.noretry=0 (check at deploy).
|
|
|
|
# Image: digest-pinned (fetch law: tag pulls silently re-point; the digest
|
|
# is what was qualified). 0.16.2 = the version the model card names as its
|
|
# quickstart pin; RepoDigest captured at pull on 2026-10-06.
|
|
FRANK_LARGE_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
|
|
# Container identity: kept from the 27B era (unit ExecStop names it).
|
|
FRANK_LARGE_CONTAINER=frank-large-halogen
|
|
# Old quadlet-era container identity; the serve script clears stale names.
|
|
FRANK_LARGE_LEGACY_CONTAINERS=frank-large
|
|
# host:container — the frontend listens on 8731 inside (HALOGEN_API_PORT
|
|
# default); the engine port (8730) is never published.
|
|
FRANK_LARGE_PUBLISH=7770:8731
|
|
|
|
FRANK_LARGE_MODEL_DIR=/home/dank/models/halogen-flashnext-v2
|
|
# The checkpoint: Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated
|
|
# qwen38-flash-next-v2-swift15-abliterated.hgn — compacted r2 of the card's
|
|
# append-only original (byte-identical live tensors, padding dropped:
|
|
# 66,687,678,464 B vs 89,775,868,032 B). LFS sha256 verified at fetch.
|
|
FRANK_LARGE_CHECKPOINT=qwen38-flash-next-v2-swift15-abliterated.hgn
|
|
FRANK_LARGE_CHECKPOINT_BYTES=66687678464
|
|
FRANK_LARGE_CHECKPOINT_SHA256=9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037
|
|
# The n-gram lookup table: REQUIRED (the engine reads it beside the
|
|
# checkpoint; the card's quickstart names it explicitly). From
|
|
# peonist-ai/halogen-qwen3.8-flash-next @ 5cc17cea.
|
|
FRANK_LARGE_NGRAM_TABLE=qwen38-flash-next-ngram.hgn
|
|
FRANK_LARGE_NGRAM_BYTES=51200246144
|
|
FRANK_LARGE_NGRAM_SHA256=3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe
|
|
# The vision tower: OPTIONAL on the engine, ENABLED here (image input).
|
|
# From peonist-ai/halogen-qwen3.8-flash-next @ e053f488.
|
|
FRANK_LARGE_VISION_TOWER=qwen38-flash-next-vision.hgn
|
|
FRANK_LARGE_VISION_BYTES=897916416
|
|
FRANK_LARGE_VISION_SHA256=d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10
|
|
# The tokenizer is a FLAT DIRECTORY, $FRANK_LARGE_MODEL_DIR/tokenizer,
|
|
# mounted whole at /tokenizer. The two repos ship the same one; fetched
|
|
# fresh from peonist @ e053f488 (tokenizer.json is LFS, byte-identical to
|
|
# the 27B-era copy; the rest are plain git blobs — size-checked only). The
|
|
# serve gate and the deployer refuse a missing file or ANY symlink: an HF
|
|
# cache snapshot symlinks each file into blobs/, and those links dangle
|
|
# inside the container mount.
|
|
FRANK_LARGE_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json"
|
|
|
|
# --- engine flags (docs/FLAGS.md of the flash-server repo; anything unset
|
|
# stays at the image default, which is what was qualified) ------------------
|
|
HALOGEN_MODEL_ID=frank-large
|
|
HALOGEN_CTX=200192
|
|
HALOGEN_KV_POOL_POSITIONS=400384
|
|
HALOGEN_KV_SLOTS=2
|
|
HALOGEN_HOST_RESERVE_GIB=20
|
|
# Spec decode: MTP head is INSIDE the checkpoint (do NOT set HALOGEN_MTP_HEAD);
|
|
# depth 2 is the documented "best all-round" default. HALOGEN_DRAFTER /
|
|
# HALOGEN_DFLASH_* from the 27B engine DO NOT EXIST here.
|
|
HALOGEN_SPEC_ADAPT=0
|
|
# Coupled pair (FLAGS.md table): the cap bounds how long ONE request holds
|
|
# the GPU, the timeout bounds how long the NEXT client waits. Do not move
|
|
# one alone (see the client-contract note above).
|
|
HALOGEN_MAX_TOK=8192
|
|
HALOGEN_MAX_TOKENS_CAP=65536
|
|
HALOGEN_QUEUE_TIMEOUT=7200
|
|
# Server-side sampler = the card's thinking recipe (the engine now HAS one;
|
|
# requests may still override per-call). Matches models.yml extraBody.
|
|
HALOGEN_TEMPERATURE=1.0
|
|
HALOGEN_TOP_P=0.95
|
|
HALOGEN_TOP_K=20
|
|
# Card's belt-and-braces: the engine pins weights by default
|
|
# (HALOGEN_FLASH_PIN_TRUNK=1; only the retired w4b build honored 0).
|
|
HALOGEN_WEIGHTS_LOCK=1
|