Frank
5c4e8a9600
Tested 16 live. Pool stayed 524288 (no clamp), server held 103.5 GiB and reported "7.1 GiB left for everything else" at BOTH 15 and 16 — identical footprint. HALOGEN_HOST_RESERVE_GIB feeds the fit-check budget only; it is not an allocation, so raising it cannot give RAM back to the co-tenants. Its sole observable effect above 16 is to take the KV pool away. 16 therefore cost half the fit margin (1.0 vs 2.0 against the engine's 36.5 GiB requirement) and bought nothing. 15 is strictly better at the same cost. Recorded the ceiling (16) and the real air levers in the unit comment: pool 262144 frees 7.2 GiB, max_tok 16384 frees 8.4.
190 lines
11 KiB
INI
190 lines
11 KiB
INI
# Quadlet unit: frank-large (rootless, boot-persistent).
|
|
#
|
|
# Install path on dankclaw: ~/.config/containers/systemd/frank-large.container
|
|
# Unit name: frank-large.service (container name: frank-large)
|
|
#
|
|
# The large inference service, since 2026-09-11 the halogen-flash-server
|
|
# runtime serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next,
|
|
# w4b .hgn checkpoint + quality overlay sidecar). It replaced the llama.cpp /
|
|
# kingjones30-ROCmFPX STRIX deployment because the engine is ~4x faster on
|
|
# prefill (the dominant cost of any real prompt) on this exact SKU.
|
|
#
|
|
# SHAPE. The image runs both halves in one container: the engine on container
|
|
# loopback (:8730, UNAUTHENTICATED — never publish it) and the OpenAI
|
|
# front-end on :8731, published here as host 7770 so the fleet's clients,
|
|
# services.json probe and docs keep the port they have always used. Same for
|
|
# HALOGEN_MODEL_ID=frank-large: it is the id at /v1/models, in /health and in
|
|
# every response, so no client had to be renamed. Halogen never rejects a
|
|
# request for naming a different model, it just serves it.
|
|
#
|
|
# DEVICES. gfx1151 through the amdgpu/KFD stack: /dev/kfd plus the DRM render
|
|
# node. Both are 0666 on this host, so the rootless user namespace can open
|
|
# them (id dank is also in render/video). seccomp=unconfined and --ipc=host are
|
|
# the image's documented requirements; memlock must be unlimited.
|
|
#
|
|
# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP
|
|
# runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
|
|
#
|
|
# MEMORY. Weights are pinned (HALOGEN_FLASH_PIN_TRUNK default 1) — that is the
|
|
# configuration the published decode numbers were measured with, and `free`,
|
|
# MemAvailable and every monitor built on them will overstate free memory by
|
|
# the size of the model (~68 GiB) because the kernel counts the locked weights
|
|
# as reclaimable page cache. Halogen prints the corrected figure at startup.
|
|
# The pinned trunk means this container holds most of the host; a co-tenant
|
|
# that then takes the remainder can stall both for minutes at full CPU with no
|
|
# disk activity and no output. That is documented behaviour, not a crash, and
|
|
# the escape hatch is HALOGEN_FLASH_PIN_TRUNK=0 (at several times the decode
|
|
# cost). Sizing is chosen for a SHARED 121 GiB host, and HALOGEN_KV_POOL_FIT
|
|
# stays on as the safety net: if the machine refuses the configured pool the
|
|
# engine lowers it itself and prints what it settled on. Read that ledger on
|
|
# every restart — it is the ground truth for sizing, not this comment.
|
|
#
|
|
# DOUBLED 2026-09-12: pool 524,288 positions (~14.4 GiB of KV at the
|
|
# measured 28 KiB/position). This did NOT fit on the first attempt. The
|
|
# engine's own ledger for the prior 262,144 config was 68.0 weights + 7.2
|
|
# pool + 21.1 working = 96.3 GiB held, 12.1 GiB left on the host, and its
|
|
# pre-flight arithmetic is: 121 GiB - 67.7 resident weights - 24.0 reserve
|
|
# = 29.5 GiB of device budget, which has to cover the pool AND the prefill
|
|
# arena together. 524,288 with the 32,768-token arena needs 35.0 GiB, so
|
|
# HALOGEN_KV_POOL_FIT lowered the pool straight back to 262,144 and the
|
|
# deploy still reported healthy. The doubling only becomes real together
|
|
# with HALOGEN_MAX_TOK=16384 further down, which halves the arena and brings
|
|
# the need to ~24.9 GiB. Read the `kv pool:` lines after every restart —
|
|
# a green health check does NOT mean the pool you asked for is the pool you got.
|
|
#
|
|
# WEIGHTS. HALOGEN_DOWNLOAD fetches the whole weights repo on FIRST START ONLY
|
|
# (the entrypoint checks whether the checkpoint is absent) into the bind mount
|
|
# below, which must therefore be read-WRITE. huggingface_hub resumes partial
|
|
# files natively, and the quadlet runs the container with --replace --rm, so a
|
|
# restart mid-fetch continues rather than starting over. Roughly 127 GB: the
|
|
# 115.55 GiB checkpoint, the 2.31 GiB quality overlay, the 0.84 GiB vision
|
|
# tower and the tokenizer. The repo is public and ungated, so no token is
|
|
# needed; later starts open no outbound connections at all.
|
|
#
|
|
# VISION. HALOGEN_VISION_TOWER=1 finds the sidecar beside the checkpoint. This
|
|
# is the fleet's only image input since the FLM NPU endpoint was retired on
|
|
# 2026-09-11; audio input has no replacement.
|
|
#
|
|
# CLIENT CONTRACT (see serverAppConfigs-docs/LLM/halogen-flash-server.md).
|
|
# There are no server-side sampler defaults: a request that sends no
|
|
# temperature decodes GREEDY. The per-request budget defaults to 8,192 tokens
|
|
# and REASONING COUNTS AGAINST IT, so a budget that runs out mid-thought
|
|
# returns finish_reason "length" with empty content and the partial reasoning
|
|
# in reasoning_content. The cap is 65,536; above it a request is a 400, never a
|
|
# silent truncation.
|
|
#
|
|
# IMAGE. The unit runs the localhost/-aliased tag: this host's podman quadlet
|
|
# path cannot resolve registry-prefixed refs from local storage at create
|
|
# time. deploy-remote.sh pulls the ghcr.io ref and re-tags it before the
|
|
# restart. Pull=never keeps boots off the registry.
|
|
[Unit]
|
|
Description=frank-large — halogen-flash-server, Qwen3.8-Flash-Next (w4b) on gfx1151
|
|
After=network-online.target
|
|
|
|
[Container]
|
|
ContainerName=frank-large
|
|
Image=localhost/peonist-ai/halogen-flash-server:0.5.8
|
|
# Published port stays 7770: Hermes, open-webui, the harness and the homepage
|
|
# probe all point here. 8730 (the engine) is deliberately not published.
|
|
PublishPort=7770:8731
|
|
AddDevice=/dev/kfd
|
|
AddDevice=/dev/dri
|
|
# Read-write: the first start downloads into it. Contains only weights.
|
|
Volume=/home/dank/models/halogen:/models
|
|
# Fetch the weights on first start only (absent checkpoint gates it).
|
|
Environment=HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next
|
|
# Fleet identity: the id at /v1/models, /health and every response body.
|
|
Environment=HALOGEN_MODEL_ID=frank-large
|
|
# Native context of the model. Matches what both clients declare.
|
|
Environment=HALOGEN_CTX=262144
|
|
# Doubled 2026-09-12 from the 262144 this had run since 09-11: two full-length
|
|
# conversations resident at once, or four at 131k, instead of one.
|
|
Environment=HALOGEN_KV_POOL_POSITIONS=524288
|
|
Environment=HALOGEN_KV_SLOTS=4
|
|
# Making the doubling stick is budget arithmetic, all of it. ONE budget covers
|
|
# the KV pool, the prefill arena and the vision tower together:
|
|
#
|
|
# budget = 121.2 host - 67.7 resident weights - reserve
|
|
# need = KV (~28 KiB/position) + arena + vision (1.97 measured)
|
|
# arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
|
|
#
|
|
# The arena term is linear in max_tok, and it reproduces five live restarts
|
|
# AND upstream's three published "on the device" figures to within 0.1 GiB —
|
|
# note those published figures are pool+arena totals, not pool alone, which is
|
|
# the misreading that first made a doubling look affordable here. Every corner
|
|
# this box offered at a reserve of 24:
|
|
#
|
|
# 16384 -> need 28.6 vs 29.5 FITS, 0.9 spare (doubling, 8-9% prefill cost)
|
|
# 32768 -> need 37.0 vs 29.5 POOL CLAMPED to 262144, deploy still green
|
|
# 65536 -> need 46.5 vs 29.5 OOM, will not boot; unit still reads ACTIVE
|
|
#
|
|
# None of those three gave both features. Dropping the reserve to 15 moves the
|
|
# budget to 38.5, so 32768's need of 37.0 fits with 1.5 spare: full prefill
|
|
# AND the doubled pool, paid for out of the n-gram page cache instead of out of
|
|
# either feature. THE TWO SETTINGS ARE A MATCHED PAIR, NOT independent tunables
|
|
# — raise the reserve back to 24 and the pool silently clamps to 262144 again.
|
|
Environment=HALOGEN_MAX_TOK=32768
|
|
# RAM left for the n-gram table the engine reads through the page cache.
|
|
# LOWERED 2026-09-12 from 24 (vendor default is 20) to buy the doubled pool at
|
|
# full prefill speed. The fit check's OWN requirement is the line to honour —
|
|
# read from a clamped boot, not modelled:
|
|
# "524288 positions need ~35.0 GiB (plus 1.5 of margin)" -> budget >= 36.5
|
|
# budget = 121.2 host - 67.7 weights - reserve
|
|
# reserve 16 -> 37.5 (1.0 spare); 15 -> 38.5 (2.0 spare, deployed); 18 -> 35.5
|
|
# CLAMPS the pool to 262144. Do not raise past 16.
|
|
#
|
|
# CEILING IS 16, AND 16 BUYS NOTHING. Tested live: 15 -> 16 changed the fit
|
|
# margin only. The server held 103.5 GiB and reported "7.1 GiB left for
|
|
# everything else" at BOTH settings, because this reserve feeds the fit-check
|
|
# budget and is not an allocation — actual footprint is weights + KV + arena +
|
|
# vision, none of which it touches. A higher reserve therefore cannot give the
|
|
# host memory back; it can only take the pool away. 15 keeps the larger margin.
|
|
#
|
|
# Measured at 15 before that test: pool reserved at 524288 with no clamp and no
|
|
# WARNING, and the server held 103.5 GiB. The ledger
|
|
# reads "host memory left for everything else: 310 contiguous 2 MiB blocks
|
|
# (7.1 GiB total)" — against 2081 blocks / 12.1 GiB at the 24-reserve
|
|
# baseline. All 28 co-tenant containers stayed up (~4.3 GiB working set).
|
|
#
|
|
# The price is that the engine volunteers this note on every boot now:
|
|
# when it runs out, BOTH they and this server can stall for minutes at a
|
|
# time at full CPU with no disk activity and no output, which is not a
|
|
# crash and needs no restart.
|
|
# That is the failure mode to watch, and it is NOT the legible one predicted
|
|
# above: a minutes-long stall that looks like a hang is harder to diagnose
|
|
# than a slow prompt. If it happens, the reserve has no headroom left to give
|
|
# — the remedy is HALOGEN_KV_POOL_POSITIONS=262144 (hands back 7.2 GiB), or
|
|
# HALOGEN_MAX_TOK=16384 (hands back 8.4) with reserve then free to rise to 24.
|
|
Environment=HALOGEN_HOST_RESERVE_GIB=15
|
|
# Image input (the sidecar ships inside the weights repo).
|
|
Environment=HALOGEN_VISION_TOWER=1
|
|
# Verbose engine logging: the vendor's first-line evidence for load,
|
|
# allocation and admission questions. Shipping it on means an incident report
|
|
# already has the full startup account instead of needing a restart-into-debug.
|
|
# HALOGEN_DMALLOC_LOG is deliberately NOT set — it logs every allocation
|
|
# over 64 MB, which is steady-state noise here; raise it only for a live
|
|
# OOM hunt.
|
|
Environment=HALOGEN_VERBOSE=1
|
|
# Required by the image: the engine maps the checkpoint and registers it with
|
|
# the GPU in place, which needs host IPC and an unconfined seccomp profile.
|
|
PodmanArgs=--ipc=host --security-opt seccomp=unconfined
|
|
Ulimit=memlock=-1:-1
|
|
# The image ships its own HEALTHCHECK; this pins it. /health asks the ENGINE
|
|
# for a PONG before answering, so 200 means "ready to serve", not "process
|
|
# alive". During the first-start download it stays unhealthy by design.
|
|
HealthCmd=halogen-healthcheck api
|
|
HealthInterval=30s
|
|
HealthTimeout=10s
|
|
HealthRetries=3
|
|
HealthStartPeriod=300s
|
|
Pull=never
|
|
|
|
[Service]
|
|
# The first start reads ~68 GiB off disk and pins it before serving.
|
|
TimeoutStartSec=900
|
|
Restart=on-failure
|
|
RestartSec=10
|
|
|
|
[Install]
|
|
WantedBy=default.target
|