Frank
41ac1f83f6
Raising HALOGEN_MAX_TOK to 65536 did not shrink the pool, it OOMed the boot:
the arena alone (37.3 GiB) exceeds the 29.5 GiB device budget, so even the
minimum pool faulted and dmalloc failed at 39.97 GiB. Rolled back to 16384,
which restored the 524288 pool and a clean completion.
The three measured arenas are exactly linear in the flag — 3.80 GiB base plus
8.40 GiB per 16384 tokens, predicting all three to 0.1 GiB. That gives a real
sizing rule instead of guesswork, and it says the deployed 16384 is the
CEILING, not a conservative choice: 29.5 - 1.5 margin - 14.4 pool leaves 13.6
for the arena, i.e. max_tok <= 19,115.
The corner check falls out of the same model: set max_tok back to 32768 and
only 7.4 GiB remains for the pool, ~269k positions, which is precisely the
262144 config this host ran before. The two corners of the feasible box are
the only two points it has ever served.
No functional change — the quadlet edit is comments only; the running service
is already at 16384 from the rollback deploy (258cccb).
165 lines
9.0 KiB
INI
165 lines
9.0 KiB
INI
# Quadlet unit: frank-large (rootless, boot-persistent).
|
|
#
|
|
# Install path on dankclaw: ~/.config/containers/systemd/frank-large.container
|
|
# Unit name: frank-large.service (container name: frank-large)
|
|
#
|
|
# The large inference service, since 2026-09-11 the halogen-flash-server
|
|
# runtime serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next,
|
|
# w4b .hgn checkpoint + quality overlay sidecar). It replaced the llama.cpp /
|
|
# kingjones30-ROCmFPX STRIX deployment because the engine is ~4x faster on
|
|
# prefill (the dominant cost of any real prompt) on this exact SKU.
|
|
#
|
|
# SHAPE. The image runs both halves in one container: the engine on container
|
|
# loopback (:8730, UNAUTHENTICATED — never publish it) and the OpenAI
|
|
# front-end on :8731, published here as host 7770 so the fleet's clients,
|
|
# services.json probe and docs keep the port they have always used. Same for
|
|
# HALOGEN_MODEL_ID=frank-large: it is the id at /v1/models, in /health and in
|
|
# every response, so no client had to be renamed. Halogen never rejects a
|
|
# request for naming a different model, it just serves it.
|
|
#
|
|
# DEVICES. gfx1151 through the amdgpu/KFD stack: /dev/kfd plus the DRM render
|
|
# node. Both are 0666 on this host, so the rootless user namespace can open
|
|
# them (id dank is also in render/video). seccomp=unconfined and --ipc=host are
|
|
# the image's documented requirements; memlock must be unlimited.
|
|
#
|
|
# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP
|
|
# runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
|
|
#
|
|
# MEMORY. Weights are pinned (HALOGEN_FLASH_PIN_TRUNK default 1) — that is the
|
|
# configuration the published decode numbers were measured with, and `free`,
|
|
# MemAvailable and every monitor built on them will overstate free memory by
|
|
# the size of the model (~68 GiB) because the kernel counts the locked weights
|
|
# as reclaimable page cache. Halogen prints the corrected figure at startup.
|
|
# The pinned trunk means this container holds most of the host; a co-tenant
|
|
# that then takes the remainder can stall both for minutes at full CPU with no
|
|
# disk activity and no output. That is documented behaviour, not a crash, and
|
|
# the escape hatch is HALOGEN_FLASH_PIN_TRUNK=0 (at several times the decode
|
|
# cost). Sizing is chosen for a SHARED 121 GiB host, and HALOGEN_KV_POOL_FIT
|
|
# stays on as the safety net: if the machine refuses the configured pool the
|
|
# engine lowers it itself and prints what it settled on. Read that ledger on
|
|
# every restart — it is the ground truth for sizing, not this comment.
|
|
#
|
|
# DOUBLED 2026-09-12: pool 524,288 positions (~14.4 GiB of KV at the
|
|
# measured 28 KiB/position). This did NOT fit on the first attempt. The
|
|
# engine's own ledger for the prior 262,144 config was 68.0 weights + 7.2
|
|
# pool + 21.1 working = 96.3 GiB held, 12.1 GiB left on the host, and its
|
|
# pre-flight arithmetic is: 121 GiB - 67.7 resident weights - 24.0 reserve
|
|
# = 29.5 GiB of device budget, which has to cover the pool AND the prefill
|
|
# arena together. 524,288 with the 32,768-token arena needs 35.0 GiB, so
|
|
# HALOGEN_KV_POOL_FIT lowered the pool straight back to 262,144 and the
|
|
# deploy still reported healthy. The doubling only becomes real together
|
|
# with HALOGEN_MAX_TOK=16384 further down, which halves the arena and brings
|
|
# the need to ~24.9 GiB. Read the `kv pool:` lines after every restart —
|
|
# a green health check does NOT mean the pool you asked for is the pool you got.
|
|
#
|
|
# WEIGHTS. HALOGEN_DOWNLOAD fetches the whole weights repo on FIRST START ONLY
|
|
# (the entrypoint checks whether the checkpoint is absent) into the bind mount
|
|
# below, which must therefore be read-WRITE. huggingface_hub resumes partial
|
|
# files natively, and the quadlet runs the container with --replace --rm, so a
|
|
# restart mid-fetch continues rather than starting over. Roughly 127 GB: the
|
|
# 115.55 GiB checkpoint, the 2.31 GiB quality overlay, the 0.84 GiB vision
|
|
# tower and the tokenizer. The repo is public and ungated, so no token is
|
|
# needed; later starts open no outbound connections at all.
|
|
#
|
|
# VISION. HALOGEN_VISION_TOWER=1 finds the sidecar beside the checkpoint. This
|
|
# is the fleet's only image input since the FLM NPU endpoint was retired on
|
|
# 2026-09-11; audio input has no replacement.
|
|
#
|
|
# CLIENT CONTRACT (see serverAppConfigs-docs/LLM/halogen-flash-server.md).
|
|
# There are no server-side sampler defaults: a request that sends no
|
|
# temperature decodes GREEDY. The per-request budget defaults to 8,192 tokens
|
|
# and REASONING COUNTS AGAINST IT, so a budget that runs out mid-thought
|
|
# returns finish_reason "length" with empty content and the partial reasoning
|
|
# in reasoning_content. The cap is 65,536; above it a request is a 400, never a
|
|
# silent truncation.
|
|
#
|
|
# IMAGE. The unit runs the localhost/-aliased tag: this host's podman quadlet
|
|
# path cannot resolve registry-prefixed refs from local storage at create
|
|
# time. deploy-remote.sh pulls the ghcr.io ref and re-tags it before the
|
|
# restart. Pull=never keeps boots off the registry.
|
|
[Unit]
|
|
Description=frank-large — halogen-flash-server, Qwen3.8-Flash-Next (w4b) on gfx1151
|
|
After=network-online.target
|
|
|
|
[Container]
|
|
ContainerName=frank-large
|
|
Image=localhost/peonist-ai/halogen-flash-server:0.5.8
|
|
# Published port stays 7770: Hermes, open-webui, the harness and the homepage
|
|
# probe all point here. 8730 (the engine) is deliberately not published.
|
|
PublishPort=7770:8731
|
|
AddDevice=/dev/kfd
|
|
AddDevice=/dev/dri
|
|
# Read-write: the first start downloads into it. Contains only weights.
|
|
Volume=/home/dank/models/halogen:/models
|
|
# Fetch the weights on first start only (absent checkpoint gates it).
|
|
Environment=HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next
|
|
# Fleet identity: the id at /v1/models, /health and every response body.
|
|
Environment=HALOGEN_MODEL_ID=frank-large
|
|
# Native context of the model. Matches what both clients declare.
|
|
Environment=HALOGEN_CTX=262144
|
|
# Doubled 2026-09-12 from 262144 to upstream's own default (2 x HALOGEN_CTX):
|
|
# two full-length conversations resident at once, or four at 131k, instead of
|
|
# one. The old pool was the binding constraint on concurrency, not CTX — a
|
|
# request reserves prompt + max_tokens against the pool, so four 65k chats
|
|
# used to contend for one 262k pool.
|
|
Environment=HALOGEN_KV_POOL_POSITIONS=524288
|
|
Environment=HALOGEN_KV_SLOTS=4
|
|
# LOAD-BEARING. This line is what makes the doubling above real; do not
|
|
# "clean it up". The device budget here is 121 GiB - 67.7 resident weights
|
|
# - 24.0 reserved for the n-gram page cache and the OS = 29.5 GiB, and that
|
|
# one budget carries BOTH the KV pool and the prefill arena.
|
|
#
|
|
# Measured across three restarts on 2026-09-12. The arena (the engine's
|
|
# "everything but the pool" figure) is linear in this flag: 3.80 GiB of base
|
|
# plus 8.40 GiB per 16384 tokens.
|
|
# 16384 -> arena 12.2, 524288 needs 26.6 -> FITS
|
|
# 32768 -> arena 20.6, 524288 needs 35.0 -> fit lowers the pool to 262144
|
|
# 65536 -> arena 37.3, even 262144 needs 44.5 -> OOM, dmalloc FAILED at 39.97
|
|
# So 16384 is not conservative, it is the ceiling: 29.5 budget - 1.5 margin -
|
|
# 14.4 pool leaves 13.6 for the arena, i.e. max_tok <= 19,115, and 16384 is
|
|
# the largest power of two under that. The reverse checks too — put max_tok
|
|
# back to 32768 and only 7.4 GiB is left for the pool, about 269k positions,
|
|
# which is exactly the 262144 config this replaced. Those two corners are the
|
|
# only points of this box that have ever run here.
|
|
#
|
|
# Do NOT raise this flag past 16384 while the pool stays doubled: 65536 does
|
|
# not shrink the pool, it stops the server booting at all — and the unit still
|
|
# reads active with the container "Up (starting)", so the health check waits
|
|
# out LLM_WAIT rather than failing fast.
|
|
Environment=HALOGEN_MAX_TOK=16384
|
|
# RAM left for the n-gram table the engine reads through the page cache.
|
|
# Above the vendor's 20 because a dozen containers share this host; too low
|
|
# and long prompts crawl with the disk busy.
|
|
Environment=HALOGEN_HOST_RESERVE_GIB=24
|
|
# Image input (the sidecar ships inside the weights repo).
|
|
Environment=HALOGEN_VISION_TOWER=1
|
|
# Verbose engine logging: the vendor's first-line evidence for load,
|
|
# allocation and admission questions. Shipping it on means an incident report
|
|
# already has the full startup account instead of needing a restart-into-debug.
|
|
# HALOGEN_DMALLOC_LOG is deliberately NOT set — it logs every allocation
|
|
# over 64 MB, which is steady-state noise here; raise it only for a live
|
|
# OOM hunt.
|
|
Environment=HALOGEN_VERBOSE=1
|
|
# Required by the image: the engine maps the checkpoint and registers it with
|
|
# the GPU in place, which needs host IPC and an unconfined seccomp profile.
|
|
PodmanArgs=--ipc=host --security-opt seccomp=unconfined
|
|
Ulimit=memlock=-1:-1
|
|
# The image ships its own HEALTHCHECK; this pins it. /health asks the ENGINE
|
|
# for a PONG before answering, so 200 means "ready to serve", not "process
|
|
# alive". During the first-start download it stays unhealthy by design.
|
|
HealthCmd=halogen-healthcheck api
|
|
HealthInterval=30s
|
|
HealthTimeout=10s
|
|
HealthRetries=3
|
|
HealthStartPeriod=300s
|
|
Pull=never
|
|
|
|
[Service]
|
|
# The first start reads ~68 GiB off disk and pins it before serving.
|
|
TimeoutStartSec=900
|
|
Restart=on-failure
|
|
RestartSec=10
|
|
|
|
[Install]
|
|
WantedBy=default.target
|