Frank
fece3ffdd0
Runtime swap ROCmFPX llama.cpp -> peonist-ai/halogen (image digest dc0a39a0, tag 0.1.4), serving qwen3.8-27b-p1w4d-d2.hgn (33.4 GiB, the .hgn format loads in no other engine) with HALOGEN_DRAFTER=2 (DFlash2) on ONE slot of 200700 ctx: slots>1 disables the drafter (engine law). Accepted: TEXT-ONLY (vision encoder ignored; frank-small owns images) and the censored base (rollback = revert this commit; payload, fork build and build-rocmfpx.sh pin stay on disk). - serve script: podman run under the root unit, exec podman wait so Restart=always is real; gates image digest, checkpoint byte-count, flat regular-file tokenizer, GPU nodes; engine stdout to log. - deploy handler: gates the artifacts, sweeps the legacy bare llama-server + old container name by NAME (never pkill llama-server: frank-small), retires the 2026-09 quadlet, port cutover 7770. - fetch script: -C - resume, -f, byte count, sha256, --verify-only. - seam rewritten (17 tests): halogen is now the deployed runtime.
104 lines
6.0 KiB
Bash
104 lines
6.0 KiB
Bash
# frank-large.env — tunables for the frank-large endpoint on Dankclaw's
|
|
# Strix Halo iGPU (gfx1151), host port 7770.
|
|
#
|
|
# RUNTIME (2026-10-05, Frank): halogen — the closed-source C++/HIP engine
|
|
# (github.com/peonist-ai/halogen-server, image ghcr.io/peonist-ai/halogen,
|
|
# digest-pinned below) serving peonist-ai/halogen-qwen3.8-27b, a ~6.32 bpw
|
|
# NVFP4-derivative quant of the BASE Qwen/Qwen3.8-27B, as a rootless
|
|
# container driven by scripts/serve-frank-large.sh. This CUTS OVER the
|
|
# 2026-10-02..10-03 ROCmFPX llama.cpp shape (uncensored
|
|
# Q6_0_ROCMFPX_AGENT + MTP, 27.6-28.1 t/s): rollback = revert the swap
|
|
# commit's trio; the ROCmFPX payload, draft head and build all stay on
|
|
# disk at /home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/.
|
|
#
|
|
# WHY THE RUNTIME MOVED WITH THE MODEL: the payload is .hgn — per its own
|
|
# card it "will not load in transformers, vLLM, or llama.cpp". Accepted
|
|
# consequences (Frank, 2026-10-05): TEXT-ONLY (halogen ignores the model's
|
|
# vision encoder; the fleet's vision path is frank-small) and the CENSORED
|
|
# base (this is a finetune of Qwen/Qwen3.8-27B, not the abliterated file —
|
|
# refusals may return). What it buys: vendor-measured decode DFlash2 31.7
|
|
# t/s mean (vs this endpoint's MTP 27.6-28.1 and the old halogen-era MTP
|
|
# 26.9), prefill ~566 t/s at 32k (was ~400 on the sibling engine), 1.4-2.3x
|
|
# wall-clock on prefill-heavy agent traffic, at HIGHER effective precision
|
|
# (~6.32 bpw vs ~6.6 with the old W4A4 promotion — see FLAGS.md).
|
|
#
|
|
# SHAPE — the two knobs that are NOT free:
|
|
# HALOGEN_KV_SLOTS=1 KEEPS SPECULATION ON. halogen makes batching and
|
|
# speculation mutually exclusive: with >1 slot the drafter is disabled
|
|
# and per-stream decode falls to serial speed (the sibling engine
|
|
# measured 41 t/s -> 14 t/s when slots were raised, and halogen's own
|
|
# FLAGS.md prints the same law). frank-large was already a one-session
|
|
# endpoint under ROCmFPX (PARALLEL=1), so slots=1 is the same contract,
|
|
# not a regression.
|
|
# HALOGEN_SLOT_CTX=200700 (Frank's window) is the per-conversation hard
|
|
# ceiling — a prompt over it is a 400 naming the limit, never a silent
|
|
# truncation. KV pool = slots x slot_ctx x 64 KiB = 12.2 GiB. Budget
|
|
# against the 121 GiB of unified memory: weights 33.4 GiB (mmap'd — page
|
|
# cache, not anon), KV 12.2, engine working set + auto prompt cache
|
|
# ~20-30, host baseline ~10 -> ~75-85 GiB in use. The 2026-09-16 halogen
|
|
# era ran the same engine class on this box with LESS headroom.
|
|
# HALOGEN_DRAFTER=2 (DFlash2) is the shipped default and the fastest
|
|
# exact drafter on this checkpoint; the .hgn BAKES BOTH drafters (MTP +
|
|
# DFlash2), so unlike the Flash-Next era there is no sidecar to mount
|
|
# and no --dflash flag — the drafter is an int knob, byte-identical
|
|
# output to greedy verification either way.
|
|
#
|
|
# CLIENT CONTRACT deltas vs the ROCmFPX shape (the harness side lives in
|
|
# ~/.omp/agent/models.yml):
|
|
# * NO server-side sampler defaults: a request without temperature
|
|
# decodes GREEDY. The harness sends temp 1.0 / top_p 0.95 via
|
|
# extraBody, so that wire recipe is load-bearing now, not cosmetic.
|
|
# * max_tokens defaults to 8192 PER REQUEST and INCLUDES thinking; a
|
|
# reasoning reply over a small budget returns empty content with
|
|
# finish_reason "length". The harness maxTokens is 57344: compaction
|
|
# fires at 70% of 200700 = 140,490 and 140,490 + 57,344 = 197,834 <=
|
|
# 200,700 keeps prompt + budget inside the pool (the 262144-era 65536
|
|
# would overflow). Requests over HALOGEN_MAX_TOKENS_CAP are a 400
|
|
# naming the cap; cap and queue timeout are COUPLED (see FLAGS.md
|
|
# table) — do not move one without the other.
|
|
# * response_format json_object/json_schema is a 400 (no constrained
|
|
# decoding in this build).
|
|
# * Thinking lands in reasoning_content — same field as the llama.cpp
|
|
# era, so the harness compat block does NOT change. reasoning_effort
|
|
# accepts none|low|xhigh on this build (template default xhigh); the
|
|
# models.yml fold (high->xhigh) still applies.
|
|
# * The engine's token protocol has no auth and binds loopback INSIDE
|
|
# the container; only the frontend port is published. Keep it that way.
|
|
|
|
# Image: digest-pinned (fetch law: tag pulls silently re-point; the digest
|
|
# is what was qualified). 0.1.4 = the version the model card names.
|
|
FRANK_LARGE_IMAGE=ghcr.io/peonist-ai/halogen@sha256:dc0a39a0016d6cfc58a197978febaafdf8d28403f724ded6111d98b5fb7ac0ea
|
|
# Container identity: the halogen container name.
|
|
FRANK_LARGE_CONTAINER=frank-large-halogen
|
|
# Old quadlet-era container identity; the serve script clears stale names.
|
|
FRANK_LARGE_LEGACY_CONTAINERS=frank-large
|
|
# host:container — the frontend listens on 8731 inside (HALOGEN_API_PORT
|
|
# default); the engine port (8730) is never published.
|
|
FRANK_LARGE_PUBLISH=7770:8731
|
|
|
|
FRANK_LARGE_MODEL_DIR=/home/dank/models/halogen-qwen3.8-27b
|
|
FRANK_LARGE_CHECKPOINT=qwen3.8-27b-p1w4d-d2.hgn
|
|
FRANK_LARGE_CHECKPOINT_BYTES=35880329102
|
|
FRANK_LARGE_CHECKPOINT_SHA256=40874e7a5c27efcf1c83c44374db85726fa4086355b3c8aae1b363644d7e6234
|
|
FRANK_LARGE_TOKENIZER=frank-large.tokenizer.gguf
|
|
FRANK_LARGE_TOKENIZER_BYTES=1293343
|
|
FRANK_LARGE_TOKENIZER_SHA256=4c1ab3a19932f7d16eefc2c7fa60715c91133fe99c6dce683a3a7f11d463e7e9
|
|
|
|
# --- engine flags (docs/FLAGS.md of the engine repo; anything unset stays
|
|
# at the image default, which is what was qualified) ------------------------
|
|
HALOGEN_MODEL_ID=frank-large
|
|
HALOGEN_SLOT_CTX=200700
|
|
HALOGEN_KV_SLOTS=1
|
|
HALOGEN_DRAFTER=2
|
|
# Coupled pair (FLAGS.md table): the cap bounds how long ONE request holds
|
|
# the GPU, the timeout bounds how long the NEXT client waits. 65536 at
|
|
# ~10 t/s serial worst case needs > 6550 s queue timeout; the defaults
|
|
# honor that. Do not lower the timeout alone.
|
|
HALOGEN_MAX_TOKENS_CAP=65536
|
|
HALOGEN_QUEUE_TIMEOUT=7200
|
|
# Engine prompt cache: EMPTY = auto (sized from available memory at
|
|
# startup). Pin an explicit MB only to argue with the auto figure — a
|
|
# full-context entry is ~18 GB at 262K, so a too-small pin yields a cache
|
|
# that reports enabled and never hits (the engine warns at startup).
|
|
#HALOGEN_CACHE_MB=
|