Frank
c489b204ea
The previous commit's doubling was a silent no-op. HALOGEN_KV_POOL_FIT measured the device budget at startup and lowered the pool straight back to 262144, while the deploy still reported healthy in 12s. The engine's arithmetic: 121 GiB host - 67.7 resident weights - 24.0 reserved for the n-gram page cache and the OS = 29.5 GiB of device budget, and that single budget carries the KV pool AND the prefill arena. At HALOGEN_MAX_TOK=32768 the arena is ~20.6 GiB, so 524288 positions need 35.0 GiB against 29.5 available. HALOGEN_MAX_TOK=16384 halves the arena to ~10.5 GiB, bringing the 524288 need to ~24.9 GiB, which fits with ~3 GiB of margin. Cost is about 8-9% of prefill throughput, per upstream. This is the engine's own prescribed fix, quoted verbatim in its refusal message. Lowering HALOGEN_CTX was the other option the engine offered; rejected because both clients declare 262144 as the context window, so it would break the client contract rather than pay for the pool.
153 lines
8.3 KiB
INI
153 lines
8.3 KiB
INI
# Quadlet unit: frank-large (rootless, boot-persistent).
|
|
#
|
|
# Install path on dankclaw: ~/.config/containers/systemd/frank-large.container
|
|
# Unit name: frank-large.service (container name: frank-large)
|
|
#
|
|
# The large inference service, since 2026-09-11 the halogen-flash-server
|
|
# runtime serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next,
|
|
# w4b .hgn checkpoint + quality overlay sidecar). It replaced the llama.cpp /
|
|
# kingjones30-ROCmFPX STRIX deployment because the engine is ~4x faster on
|
|
# prefill (the dominant cost of any real prompt) on this exact SKU.
|
|
#
|
|
# SHAPE. The image runs both halves in one container: the engine on container
|
|
# loopback (:8730, UNAUTHENTICATED — never publish it) and the OpenAI
|
|
# front-end on :8731, published here as host 7770 so the fleet's clients,
|
|
# services.json probe and docs keep the port they have always used. Same for
|
|
# HALOGEN_MODEL_ID=frank-large: it is the id at /v1/models, in /health and in
|
|
# every response, so no client had to be renamed. Halogen never rejects a
|
|
# request for naming a different model, it just serves it.
|
|
#
|
|
# DEVICES. gfx1151 through the amdgpu/KFD stack: /dev/kfd plus the DRM render
|
|
# node. Both are 0666 on this host, so the rootless user namespace can open
|
|
# them (id dank is also in render/video). seccomp=unconfined and --ipc=host are
|
|
# the image's documented requirements; memlock must be unlimited.
|
|
#
|
|
# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP
|
|
# runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
|
|
#
|
|
# MEMORY. Weights are pinned (HALOGEN_FLASH_PIN_TRUNK default 1) — that is the
|
|
# configuration the published decode numbers were measured with, and `free`,
|
|
# MemAvailable and every monitor built on them will overstate free memory by
|
|
# the size of the model (~68 GiB) because the kernel counts the locked weights
|
|
# as reclaimable page cache. Halogen prints the corrected figure at startup.
|
|
# The pinned trunk means this container holds most of the host; a co-tenant
|
|
# that then takes the remainder can stall both for minutes at full CPU with no
|
|
# disk activity and no output. That is documented behaviour, not a crash, and
|
|
# the escape hatch is HALOGEN_FLASH_PIN_TRUNK=0 (at several times the decode
|
|
# cost). Sizing is chosen for a SHARED 121 GiB host, and HALOGEN_KV_POOL_FIT
|
|
# stays on as the safety net: if the machine refuses the configured pool the
|
|
# engine lowers it itself and prints what it settled on. Read that ledger on
|
|
# every restart — it is the ground truth for sizing, not this comment.
|
|
#
|
|
# DOUBLED 2026-09-12: pool 524,288 positions (~14.4 GiB of KV at the
|
|
# measured 28 KiB/position). This did NOT fit on the first attempt. The
|
|
# engine's own ledger for the prior 262,144 config was 68.0 weights + 7.2
|
|
# pool + 21.1 working = 96.3 GiB held, 12.1 GiB left on the host, and its
|
|
# pre-flight arithmetic is: 121 GiB - 67.7 resident weights - 24.0 reserve
|
|
# = 29.5 GiB of device budget, which has to cover the pool AND the prefill
|
|
# arena together. 524,288 with the 32,768-token arena needs 35.0 GiB, so
|
|
# HALOGEN_KV_POOL_FIT lowered the pool straight back to 262,144 and the
|
|
# deploy still reported healthy. The doubling only becomes real together
|
|
# with HALOGEN_MAX_TOK=16384 further down, which halves the arena and brings
|
|
# the need to ~24.9 GiB. Read the `kv pool:` lines after every restart —
|
|
# a green health check does NOT mean the pool you asked for is the pool you got.
|
|
#
|
|
# WEIGHTS. HALOGEN_DOWNLOAD fetches the whole weights repo on FIRST START ONLY
|
|
# (the entrypoint checks whether the checkpoint is absent) into the bind mount
|
|
# below, which must therefore be read-WRITE. huggingface_hub resumes partial
|
|
# files natively, and the quadlet runs the container with --replace --rm, so a
|
|
# restart mid-fetch continues rather than starting over. Roughly 127 GB: the
|
|
# 115.55 GiB checkpoint, the 2.31 GiB quality overlay, the 0.84 GiB vision
|
|
# tower and the tokenizer. The repo is public and ungated, so no token is
|
|
# needed; later starts open no outbound connections at all.
|
|
#
|
|
# VISION. HALOGEN_VISION_TOWER=1 finds the sidecar beside the checkpoint. This
|
|
# is the fleet's only image input since the FLM NPU endpoint was retired on
|
|
# 2026-09-11; audio input has no replacement.
|
|
#
|
|
# CLIENT CONTRACT (see serverAppConfigs-docs/LLM/halogen-flash-server.md).
|
|
# There are no server-side sampler defaults: a request that sends no
|
|
# temperature decodes GREEDY. The per-request budget defaults to 8,192 tokens
|
|
# and REASONING COUNTS AGAINST IT, so a budget that runs out mid-thought
|
|
# returns finish_reason "length" with empty content and the partial reasoning
|
|
# in reasoning_content. The cap is 65,536; above it a request is a 400, never a
|
|
# silent truncation.
|
|
#
|
|
# IMAGE. The unit runs the localhost/-aliased tag: this host's podman quadlet
|
|
# path cannot resolve registry-prefixed refs from local storage at create
|
|
# time. deploy-remote.sh pulls the ghcr.io ref and re-tags it before the
|
|
# restart. Pull=never keeps boots off the registry.
|
|
[Unit]
|
|
Description=frank-large — halogen-flash-server, Qwen3.8-Flash-Next (w4b) on gfx1151
|
|
After=network-online.target
|
|
|
|
[Container]
|
|
ContainerName=frank-large
|
|
Image=localhost/peonist-ai/halogen-flash-server:0.5.8
|
|
# Published port stays 7770: Hermes, open-webui, the harness and the homepage
|
|
# probe all point here. 8730 (the engine) is deliberately not published.
|
|
PublishPort=7770:8731
|
|
AddDevice=/dev/kfd
|
|
AddDevice=/dev/dri
|
|
# Read-write: the first start downloads into it. Contains only weights.
|
|
Volume=/home/dank/models/halogen:/models
|
|
# Fetch the weights on first start only (absent checkpoint gates it).
|
|
Environment=HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next
|
|
# Fleet identity: the id at /v1/models, /health and every response body.
|
|
Environment=HALOGEN_MODEL_ID=frank-large
|
|
# Native context of the model. Matches what both clients declare.
|
|
Environment=HALOGEN_CTX=262144
|
|
# Doubled 2026-09-12 from 262144 to upstream's own default (2 x HALOGEN_CTX):
|
|
# two full-length conversations resident at once, or four at 131k, instead of
|
|
# one. The old pool was the binding constraint on concurrency, not CTX — a
|
|
# request reserves prompt + max_tokens against the pool, so four 65k chats
|
|
# used to contend for one 262k pool.
|
|
Environment=HALOGEN_KV_POOL_POSITIONS=524288
|
|
Environment=HALOGEN_KV_SLOTS=4
|
|
# LOAD-BEARING. This line is what makes the doubling above real; do not
|
|
# "clean it up". The device budget here is 121 GiB - 67.7 resident weights
|
|
# - 24.0 reserved for the n-gram page cache and the OS = 29.5 GiB, and that
|
|
# one budget carries BOTH the KV pool and the prefill arena. At the upstream
|
|
# HALOGEN_MAX_TOK=32768 the arena is ~20.6 GiB, so 524288 positions need
|
|
# 35.0 GiB against 29.5 available and HALOGEN_KV_POOL_FIT silently lowers
|
|
# the pool back to 262144 — the deploy still reports healthy, so the
|
|
# regression is invisible unless you read the ledger. Halving the arena to
|
|
# ~10.5 GiB brings the 524288 need to ~24.9 GiB, which fits with ~3 GiB
|
|
# of margin, at about 8-9% of prefill throughput.
|
|
Environment=HALOGEN_MAX_TOK=16384
|
|
# RAM left for the n-gram table the engine reads through the page cache.
|
|
# Above the vendor's 20 because a dozen containers share this host; too low
|
|
# and long prompts crawl with the disk busy.
|
|
Environment=HALOGEN_HOST_RESERVE_GIB=24
|
|
# Image input (the sidecar ships inside the weights repo).
|
|
Environment=HALOGEN_VISION_TOWER=1
|
|
# Verbose engine logging: the vendor's first-line evidence for load,
|
|
# allocation and admission questions. Shipping it on means an incident report
|
|
# already has the full startup account instead of needing a restart-into-debug.
|
|
# HALOGEN_DMALLOC_LOG is deliberately NOT set — it logs every allocation
|
|
# over 64 MB, which is steady-state noise here; raise it only for a live
|
|
# OOM hunt.
|
|
Environment=HALOGEN_VERBOSE=1
|
|
# Required by the image: the engine maps the checkpoint and registers it with
|
|
# the GPU in place, which needs host IPC and an unconfined seccomp profile.
|
|
PodmanArgs=--ipc=host --security-opt seccomp=unconfined
|
|
Ulimit=memlock=-1:-1
|
|
# The image ships its own HEALTHCHECK; this pins it. /health asks the ENGINE
|
|
# for a PONG before answering, so 200 means "ready to serve", not "process
|
|
# alive". During the first-start download it stays unhealthy by design.
|
|
HealthCmd=halogen-healthcheck api
|
|
HealthInterval=30s
|
|
HealthTimeout=10s
|
|
HealthRetries=3
|
|
HealthStartPeriod=300s
|
|
Pull=never
|
|
|
|
[Service]
|
|
# The first start reads ~68 GiB off disk and pins it before serving.
|
|
TimeoutStartSec=900
|
|
Restart=on-failure
|
|
RestartSec=10
|
|
|
|
[Install]
|
|
WantedBy=default.target
|