Frank
bdbe814fac
The first constants were wrong in three ways, found while resuming the checkpoint (the file was COMPLETE; the manifest was not): - checkpoint is 35,865,565,184 B / LFS sha256 274c3fc7... (not 35,880,329,102 / 40874e7a...), 1352 tensors per the card; - the tokenizer is a FLAT tokenizer/ DIRECTORY of 5 files mounted whole at /tokenizer (model-card docker-run parity; the engine's HALOGEN_TOKENIZER default is the directory) — there is no frank-large.tokenizer.gguf; - plain git blobs (vocab/merges/jinja/config) carry no LFS oid, so the fetch script size-checks them and sha-verifies only the two LFS objects (checkpoint, tokenizer.json). Gates flipped together (serve + deployer twins): tokenizer dir must be a real directory and every named file a regular file, no symlinks (HF snapshot links dangle in the mount). Fetch made idempotent: a file at the published size skips the transfer (resuming it is a CDN 416 under set -e). Seam rewritten to the same contract, 17/17.
114 lines
6.7 KiB
Bash
114 lines
6.7 KiB
Bash
# frank-large.env — tunables for the frank-large endpoint on Dankclaw's
|
|
# Strix Halo iGPU (gfx1151), host port 7770.
|
|
#
|
|
# RUNTIME (2026-10-05, Frank): halogen — the closed-source C++/HIP engine
|
|
# (github.com/peonist-ai/halogen-server, image ghcr.io/peonist-ai/halogen,
|
|
# digest-pinned below) serving peonist-ai/halogen-qwen3.8-27b, a ~6.32 bpw
|
|
# NVFP4-derivative quant of the BASE Qwen/Qwen3.8-27B (35,865,565,184 B,
|
|
# 1352 tensors), as a podman container under a root unit, driven by
|
|
# scripts/serve-frank-large.sh. This CUTS OVER the
|
|
# 2026-10-02..10-03 ROCmFPX llama.cpp shape (uncensored
|
|
# Q6_0_ROCMFPX_AGENT + MTP, 27.6-28.1 t/s): rollback = revert the swap
|
|
# commit's trio; the ROCmFPX payload, draft head and build all stay on
|
|
# disk at /home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/.
|
|
#
|
|
# WHY THE RUNTIME MOVED WITH THE MODEL: the payload is .hgn — per its own
|
|
# card it "will not load in transformers, vLLM, or llama.cpp". Accepted
|
|
# consequences (Frank, 2026-10-05): TEXT-ONLY (halogen ignores the model's
|
|
# vision encoder; the fleet's vision path is frank-small) and the CENSORED
|
|
# base (this is a finetune of Qwen/Qwen3.8-27B, not the abliterated file —
|
|
# refusals may return). What it buys: vendor-measured decode DFlash2 31.7
|
|
# t/s mean (vs this endpoint's MTP 27.6-28.1 and the old halogen-era MTP
|
|
# 26.9), prefill ~566 t/s at 32k (was ~400 on the sibling engine), 1.4-2.3x
|
|
# wall-clock on prefill-heavy agent traffic, at HIGHER effective precision
|
|
# (~6.32 bpw vs ~6.6 with the old W4A4 promotion — see FLAGS.md).
|
|
#
|
|
# SHAPE — the two knobs that are NOT free:
|
|
# HALOGEN_KV_SLOTS=1 KEEPS SPECULATION ON. halogen makes batching and
|
|
# speculation mutually exclusive: with >1 slot the drafter is disabled
|
|
# and per-stream decode falls to serial speed (the sibling engine
|
|
# measured 41 t/s -> 14 t/s when slots were raised, and halogen's own
|
|
# FLAGS.md prints the same law). frank-large was already a one-session
|
|
# endpoint under ROCmFPX (PARALLEL=1), so slots=1 is the same contract,
|
|
# not a regression.
|
|
# HALOGEN_SLOT_CTX=200700 (Frank's window) is the per-conversation hard
|
|
# ceiling — a prompt over it is a 400 naming the limit, never a silent
|
|
# truncation. KV pool = slots x slot_ctx x 64 KiB = 12.2 GiB. Budget
|
|
# against the 121 GiB of unified memory: weights 33.4 GiB (mmap'd — page
|
|
# cache, not anon), KV 12.2, engine working set + auto prompt cache
|
|
# ~20-30, host baseline ~10 -> ~75-85 GiB in use. The 2026-09-16 halogen
|
|
# era ran the same engine class on this box with LESS headroom.
|
|
# HALOGEN_DRAFTER=2 (DFlash2) is the shipped default and the fastest
|
|
# exact drafter on this checkpoint; the .hgn BAKES BOTH drafters (MTP +
|
|
# DFlash2), so unlike the Flash-Next era there is no sidecar to mount
|
|
# and no --dflash flag — the drafter is an int knob, byte-identical
|
|
# output to greedy verification either way.
|
|
#
|
|
# CLIENT CONTRACT deltas vs the ROCmFPX shape (the harness side lives in
|
|
# ~/.omp/agent/models.yml):
|
|
# * NO server-side sampler defaults: a request without temperature
|
|
# decodes GREEDY. The harness sends temp 1.0 / top_p 0.95 via
|
|
# extraBody, so that wire recipe is load-bearing now, not cosmetic.
|
|
# * max_tokens defaults to 8192 PER REQUEST and INCLUDES thinking; a
|
|
# reasoning reply over a small budget returns empty content with
|
|
# finish_reason "length". The harness maxTokens is 57344: compaction
|
|
# fires at 70% of 200700 = 140,490 and 140,490 + 57,344 = 197,834 <=
|
|
# 200,700 keeps prompt + budget inside the pool (the 262144-era 65536
|
|
# would overflow). Requests over HALOGEN_MAX_TOKENS_CAP are a 400
|
|
# naming the cap; cap and queue timeout are COUPLED (see FLAGS.md
|
|
# table) — do not move one without the other.
|
|
# * response_format json_object/json_schema is a 400 (no constrained
|
|
# decoding in this build).
|
|
# * Thinking lands in reasoning_content — same field as the llama.cpp
|
|
# era, so the harness compat block does NOT change. reasoning_effort
|
|
# accepts none|low|xhigh on this build (template default xhigh); the
|
|
# models.yml fold (high->xhigh) still applies.
|
|
# * The engine's token protocol has no auth and binds loopback INSIDE
|
|
# the container; only the frontend port is published. Keep it that way.
|
|
|
|
# Image: digest-pinned (fetch law: tag pulls silently re-point; the digest
|
|
# is what was qualified). 0.1.4 = the version the model card names.
|
|
FRANK_LARGE_IMAGE=ghcr.io/peonist-ai/halogen@sha256:dc0a39a0016d6cfc58a197978febaafdf8d28403f724ded6111d98b5fb7ac0ea
|
|
# Container identity: the halogen container name.
|
|
FRANK_LARGE_CONTAINER=frank-large-halogen
|
|
# Old quadlet-era container identity; the serve script clears stale names.
|
|
FRANK_LARGE_LEGACY_CONTAINERS=frank-large
|
|
# host:container — the frontend listens on 8731 inside (HALOGEN_API_PORT
|
|
# default); the engine port (8730) is never published.
|
|
FRANK_LARGE_PUBLISH=7770:8731
|
|
|
|
FRANK_LARGE_MODEL_DIR=/home/dank/models/halogen-qwen3.8-27b
|
|
FRANK_LARGE_CHECKPOINT=qwen3.8-27b-p1w4d-d2.hgn
|
|
FRANK_LARGE_CHECKPOINT_BYTES=35865565184
|
|
FRANK_LARGE_CHECKPOINT_SHA256=274c3fc767fb57faf025dd03c76376b1fc1b32448aef45aaf02d1898fa962089
|
|
# The tokenizer is a FLAT DIRECTORY, $FRANK_LARGE_MODEL_DIR/tokenizer, mounted
|
|
# whole at /tokenizer — the model card ships `tokenizer/` and the engine's
|
|
# HALOGEN_TOKENIZER default is that directory (the card's own docker run sets
|
|
# no tokenizer flag). The Hub stores it as 5 files (sizes from the Hub API,
|
|
# 2026-10-05): tokenizer.json 12809320 (LFS sha256
|
|
# 0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3),
|
|
# vocab.json 6722759, merges.txt 3353259, chat_template.jinja 8952,
|
|
# tokenizer_config.json 17928 (plain git blobs — size-checked, no LFS oid).
|
|
# The serve gate and the deployer refuse a missing file or ANY symlink: an
|
|
# HF cache snapshot symlinks each file into blobs/, and those links dangle
|
|
# inside the container mount.
|
|
FRANK_LARGE_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json"
|
|
|
|
# --- engine flags (docs/FLAGS.md of the engine repo; anything unset stays
|
|
# at the image default, which is what was qualified) ------------------------
|
|
HALOGEN_MODEL_ID=frank-large
|
|
HALOGEN_SLOT_CTX=200700
|
|
HALOGEN_KV_SLOTS=1
|
|
HALOGEN_DRAFTER=2
|
|
# Coupled pair (FLAGS.md table): the cap bounds how long ONE request holds
|
|
# the GPU, the timeout bounds how long the NEXT client waits. 65536 at
|
|
# ~10 t/s serial worst case needs > 6550 s queue timeout; the defaults
|
|
# honor that. Do not lower the timeout alone.
|
|
HALOGEN_MAX_TOKENS_CAP=65536
|
|
HALOGEN_QUEUE_TIMEOUT=7200
|
|
# Engine prompt cache: EMPTY = auto (sized from available memory at
|
|
# startup). Pin an explicit MB only to argue with the auto figure — a
|
|
# full-context entry is ~18 GB at 262K, so a too-small pin yields a cache
|
|
# that reports enabled and never hits (the engine warns at startup).
|
|
#HALOGEN_CACHE_MB=
|