Frank
ba67ca5c9e
Auto cache-sizing is OFF on this box: the boot ledger refused it at 36.9 GB (2 x full-entry) floor vs 35.7 GB free, so every deep agent request re-prefills its whole 79-81k context (84.9-179.8 s measured on serve_api lines, 2026-10-06) for ~500-650 new tokens per turn. Pin HALOGEN_CACHE_MB=16384 (>= one full-slot entry at ~70 KB/token) — the documented override; the per-store reserve re-check keeps the host safe. Also: the serve script's 'podman run -d > LOG' captured only the container ID — the runbook's boot-ledger grep read a one-line hex file and the ledger existed nowhere on disk. Foreground 'podman run' makes the ledger land in the log file as documented; exit-code and ExecStop semantics are unchanged. Test seam updated for both.
128 lines
7.6 KiB
Bash
128 lines
7.6 KiB
Bash
# frank-large.env — tunables for the frank-large endpoint on Dankclaw's
|
|
# Strix Halo iGPU (gfx1151), host port 7770.
|
|
#
|
|
# RUNTIME (2026-10-05, Frank): halogen — the closed-source C++/HIP engine
|
|
# (github.com/peonist-ai/halogen-server, image ghcr.io/peonist-ai/halogen,
|
|
# digest-pinned below) serving peonist-ai/halogen-qwen3.8-27b, a ~6.32 bpw
|
|
# NVFP4-derivative quant of the BASE Qwen/Qwen3.8-27B (35,865,565,184 B,
|
|
# 1352 tensors), as a podman container under a root unit, driven by
|
|
# scripts/serve-frank-large.sh. This CUTS OVER the
|
|
# 2026-10-02..10-03 ROCmFPX llama.cpp shape (uncensored
|
|
# Q6_0_ROCMFPX_AGENT + MTP, 27.6-28.1 t/s): rollback = revert the swap
|
|
# commit's trio; the ROCmFPX payload, draft head and build all stay on
|
|
# disk at /home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/.
|
|
#
|
|
# WHY THE RUNTIME MOVED WITH THE MODEL: the payload is .hgn — per its own
|
|
# card it "will not load in transformers, vLLM, or llama.cpp". Accepted
|
|
# consequences (Frank, 2026-10-05): TEXT-ONLY (halogen ignores the model's
|
|
# vision encoder; the fleet's vision path is frank-small) and the CENSORED
|
|
# base (this is a finetune of Qwen/Qwen3.8-27B, not the abliterated file —
|
|
# refusals may return). What it buys: vendor-measured decode DFlash2 31.7
|
|
# t/s mean (vs this endpoint's MTP 27.6-28.1 and the old halogen-era MTP
|
|
# 26.9), prefill ~566 t/s at 32k (was ~400 on the sibling engine), 1.4-2.3x
|
|
# wall-clock on prefill-heavy agent traffic, at HIGHER effective precision
|
|
# (~6.32 bpw vs ~6.6 with the old W4A4 promotion — see FLAGS.md).
|
|
#
|
|
# SHAPE — the two knobs that are NOT free:
|
|
# HALOGEN_KV_SLOTS=1 KEEPS SPECULATION ON. halogen makes batching and
|
|
# speculation mutually exclusive: with >1 slot the drafter is disabled
|
|
# and per-stream decode falls to serial speed (the sibling engine
|
|
# measured 41 t/s -> 14 t/s when slots were raised, and halogen's own
|
|
# FLAGS.md prints the same law). frank-large was already a one-session
|
|
# endpoint under ROCmFPX (PARALLEL=1), so slots=1 is the same contract,
|
|
# not a regression.
|
|
# HALOGEN_SLOT_CTX=200700 (Frank's window) is the per-conversation hard
|
|
# ceiling — a prompt over it is a 400 naming the limit, never a silent
|
|
# truncation. KV pool = slots x slot_ctx x 64 KiB = 12.2 GiB. Budget
|
|
# against the 121 GiB of unified memory: weights 33.4 GiB (mmap'd — page
|
|
# cache, not anon), KV 12.2, engine working set + auto prompt cache
|
|
# ~20-30, host baseline ~10 -> ~75-85 GiB in use. The 2026-09-16 halogen
|
|
# era ran the same engine class on this box with LESS headroom.
|
|
# HALOGEN_DRAFTER=2 (DFlash2) is the shipped default and the fastest
|
|
# exact drafter on this checkpoint; the .hgn BAKES BOTH drafters (MTP +
|
|
# DFlash2), so unlike the Flash-Next era there is no sidecar to mount
|
|
# and no --dflash flag — the drafter is an int knob, byte-identical
|
|
# output to greedy verification either way.
|
|
#
|
|
# CLIENT CONTRACT deltas vs the ROCmFPX shape (the harness side lives in
|
|
# ~/.omp/agent/models.yml):
|
|
# * NO server-side sampler defaults: a request without temperature
|
|
# decodes GREEDY. The harness sends temp 1.0 / top_p 0.95 via
|
|
# extraBody, so that wire recipe is load-bearing now, not cosmetic.
|
|
# * max_tokens defaults to 8192 PER REQUEST and INCLUDES thinking; a
|
|
# reasoning reply over a small budget returns empty content with
|
|
# finish_reason "length". The harness maxTokens is 57344: compaction
|
|
# fires at 70% of 200700 = 140,490 and 140,490 + 57,344 = 197,834 <=
|
|
# 200,700 keeps prompt + budget inside the pool (the 262144-era 65536
|
|
# would overflow). Requests over HALOGEN_MAX_TOKENS_CAP are a 400
|
|
# naming the cap; cap and queue timeout are COUPLED (see FLAGS.md
|
|
# table) — do not move one without the other.
|
|
# * response_format json_object/json_schema is a 400 (no constrained
|
|
# decoding in this build).
|
|
# * Thinking lands in reasoning_content — same field as the llama.cpp
|
|
# era, so the harness compat block does NOT change. reasoning_effort
|
|
# accepts none|low|xhigh on this build (template default xhigh); the
|
|
# models.yml fold (high->xhigh) still applies.
|
|
# * The engine's token protocol has no auth and binds loopback INSIDE
|
|
# the container; only the frontend port is published. Keep it that way.
|
|
|
|
# Image: digest-pinned (fetch law: tag pulls silently re-point; the digest
|
|
# is what was qualified). 0.1.4 = the version the model card names.
|
|
FRANK_LARGE_IMAGE=ghcr.io/peonist-ai/halogen@sha256:dc0a39a0016d6cfc58a197978febaafdf8d28403f724ded6111d98b5fb7ac0ea
|
|
# Container identity: the halogen container name.
|
|
FRANK_LARGE_CONTAINER=frank-large-halogen
|
|
# Old quadlet-era container identity; the serve script clears stale names.
|
|
FRANK_LARGE_LEGACY_CONTAINERS=frank-large
|
|
# host:container — the frontend listens on 8731 inside (HALOGEN_API_PORT
|
|
# default); the engine port (8730) is never published.
|
|
FRANK_LARGE_PUBLISH=7770:8731
|
|
|
|
FRANK_LARGE_MODEL_DIR=/home/dank/models/halogen-qwen3.8-27b
|
|
FRANK_LARGE_CHECKPOINT=qwen3.8-27b-p1w4d-d2.hgn
|
|
FRANK_LARGE_CHECKPOINT_BYTES=35865565184
|
|
FRANK_LARGE_CHECKPOINT_SHA256=274c3fc767fb57faf025dd03c76376b1fc1b32448aef45aaf02d1898fa962089
|
|
# The tokenizer is a FLAT DIRECTORY, $FRANK_LARGE_MODEL_DIR/tokenizer, mounted
|
|
# whole at /tokenizer — the model card ships `tokenizer/` and the engine's
|
|
# HALOGEN_TOKENIZER default is that directory (the card's own docker run sets
|
|
# no tokenizer flag). The Hub stores it as 5 files (sizes from the Hub API,
|
|
# 2026-10-05): tokenizer.json 12809320 (LFS sha256
|
|
# 0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3),
|
|
# vocab.json 6722759, merges.txt 3353259, chat_template.jinja 8952,
|
|
# tokenizer_config.json 17928 (plain git blobs — size-checked, no LFS oid).
|
|
# The serve gate and the deployer refuse a missing file or ANY symlink: an
|
|
# HF cache snapshot symlinks each file into blobs/, and those links dangle
|
|
# inside the container mount.
|
|
FRANK_LARGE_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json"
|
|
|
|
# --- engine flags (docs/FLAGS.md of the engine repo; anything unset stays
|
|
# at the image default, which is what was qualified) ------------------------
|
|
HALOGEN_MODEL_ID=frank-large
|
|
HALOGEN_SLOT_CTX=200700
|
|
HALOGEN_KV_SLOTS=1
|
|
HALOGEN_DRAFTER=2
|
|
# Coupled pair (FLAGS.md table): the cap bounds how long ONE request holds
|
|
# the GPU, the timeout bounds how long the NEXT client waits. 65536 at
|
|
# ~10 t/s serial worst case needs > 6550 s queue timeout; the defaults
|
|
# honor that. Do not lower the timeout alone.
|
|
HALOGEN_MAX_TOKENS_CAP=65536
|
|
HALOGEN_QUEUE_TIMEOUT=7200
|
|
# Prompt cache — PINNED (2026-10-06). Auto is OFF on this box: the boot
|
|
# ledger refused it — "serve: prompt cache OFF - 35.7 GB free after the
|
|
# mapped checkpoint, 8.6 GB reserved for the system, which leaves less
|
|
# than the 36.9 GB floor (2 x a 18.4 GB full-context entry). Set
|
|
# HALOGEN_CACHE_MB to override." Auto demands TWO full-context entries;
|
|
# the documented override is an explicit byte budget. Measured cost of
|
|
# leaving it off: every deep agent request re-prefills its whole
|
|
# context — 84.9-179.8 s for 46-81k prompts that grow by only ~500-650
|
|
# tokens per turn (serve_api lines, 2026-10-06) — while the README
|
|
# prices the cache at ~20x time-to-first-token on a warm turn.
|
|
#
|
|
# Sizing: an entry costs ~70 KB/token (engine figure), so ONE full-slot
|
|
# entry (200,700 tok) is ~13.7 GB; 16384 MB holds it with margin, and
|
|
# several ordinary 60-80k conversations under LRU-by-bytes eviction.
|
|
# The reserve (HALOGEN_CACHE_RESERVE_MB, default 8192) is re-checked
|
|
# against live MemAvailable on every store, so a pin cannot starve the
|
|
# host silently. HALOGEN_CACHE_ALIGN stays at its shipped 2048 — that
|
|
# value is what makes warm answers byte-identical to cold ones.
|
|
HALOGEN_CACHE_MB=16384
|