Frank
27e63359d6
Predicted ~10.8 GiB KV / ~33.4 need / ~10.5 host-left; the boot ledger printed ~11.0 / ~31.5 / 100.0 GiB held with 9.1 GiB + 1396 contiguous 2 MiB blocks left. Stall NOTE no longer prints at this footprint. Also documents the ~45-60 s first-request PING silence after a restart (the healthcheck's HealthStartPeriod covers it; do not restart the unit).
210 lines
12 KiB
INI
210 lines
12 KiB
INI
# Quadlet unit: frank-large (rootless, boot-persistent).
|
|
#
|
|
# Install path on dankclaw: ~/.config/containers/systemd/frank-large.container
|
|
# Unit name: frank-large.service (container name: frank-large)
|
|
#
|
|
# The large inference service, since 2026-09-11 the halogen-flash-server
|
|
# runtime serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next,
|
|
# w4b .hgn checkpoint + quality overlay sidecar). It replaced the llama.cpp /
|
|
# kingjones30-ROCmFPX STRIX deployment because the engine is ~4x faster on
|
|
# prefill (the dominant cost of any real prompt) on this exact SKU.
|
|
#
|
|
# SHAPE. The image runs both halves in one container: the engine on container
|
|
# loopback (:8730, UNAUTHENTICATED — never publish it) and the OpenAI
|
|
# front-end on :8731, published here as host 7770 so the fleet's clients,
|
|
# services.json probe and docs keep the port they have always used. Same for
|
|
# HALOGEN_MODEL_ID=frank-large: it is the id at /v1/models, in /health and in
|
|
# every response, so no client had to be renamed. Halogen never rejects a
|
|
# request for naming a different model, it just serves it.
|
|
#
|
|
# DEVICES. gfx1151 through the amdgpu/KFD stack: /dev/kfd plus the DRM render
|
|
# node. Both are 0666 on this host, so the rootless user namespace can open
|
|
# them (id dank is also in render/video). seccomp=unconfined and --ipc=host are
|
|
# the image's documented requirements; memlock must be unlimited.
|
|
#
|
|
# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP
|
|
# runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
|
|
#
|
|
# MEMORY. Weights are pinned (HALOGEN_FLASH_PIN_TRUNK default 1) — that is the
|
|
# configuration the published decode numbers were measured with, and `free`,
|
|
# MemAvailable and every monitor built on them will overstate free memory by
|
|
# the size of the model (~68 GiB) because the kernel counts the locked weights
|
|
# as reclaimable page cache. Halogen prints the corrected figure at startup.
|
|
# The pinned trunk means this container holds most of the host; a co-tenant
|
|
# that then takes the remainder can stall both for minutes at full CPU with no
|
|
# disk activity and no output. That is documented behaviour, not a crash, and
|
|
# the escape hatch is HALOGEN_FLASH_PIN_TRUNK=0 (at several times the decode
|
|
# cost). Sizing is chosen for a SHARED 121 GiB host, and HALOGEN_KV_POOL_FIT
|
|
# stays on as the safety net: if the machine refuses the configured pool the
|
|
# engine lowers it itself and prints what it settled on. Read that ledger on
|
|
# every restart — it is the ground truth for sizing, not this comment.
|
|
#
|
|
# POOL TRIMMED 2026-09-13: 401,408 positions (~11.0 GiB of KV as reserved by
|
|
# the engine), down from the 524,288 the 2026-09-12 doubling deployed. The
|
|
# doubling fit only at reserve 15, which left the host 7.1 GiB / 310
|
|
# contiguous 2 MiB blocks and made the engine print its co-tenant stall
|
|
# warning on every boot. MEASURED AT 401,408 (deploy ledger): 68.0 weights +
|
|
# 11.0 KV + 21.0 working = 100.0 GiB held, 9.1 GiB / 1396 contiguous 2 MiB
|
|
# blocks left, engine need printed ~31.5 vs the modelled ~33.4, and NO
|
|
# stall-warning NOTE. Capacity: one full 262k conversation resident beside a
|
|
# ~139k one, or three concurrent 131k chats -- the fleet's real shape at
|
|
# three slots. Fit check budget: 121.2 - 67.7 - 15 = 38.5. The 09-12 doubling
|
|
# story (silent clamp at reserve 24, the arena term, the reserve ceiling) is
|
|
# preserved in the budget block below and the runbook.
|
|
#
|
|
# BOOT NOTE: on the first request after a (re)start the engine can go silent
|
|
# for ~45-60 s ("has not answered PING") while it first-touches the freshly
|
|
# reserved pool; /health reads engine_unresponsive during it and recovers by
|
|
# itself. The quadlet health check's HealthStartPeriod=300s covers it; do not
|
|
# restart the unit over it.
|
|
#
|
|
# WEIGHTS. HALOGEN_DOWNLOAD fetches the whole weights repo on FIRST START ONLY
|
|
# (the entrypoint checks whether the checkpoint is absent) into the bind mount
|
|
# below, which must therefore be read-WRITE. huggingface_hub resumes partial
|
|
# files natively, and the quadlet runs the container with --replace --rm, so a
|
|
# restart mid-fetch continues rather than starting over. Roughly 127 GB: the
|
|
# 115.55 GiB checkpoint, the 2.31 GiB quality overlay, the 0.84 GiB vision
|
|
# tower and the tokenizer. The repo is public and ungated, so no token is
|
|
# needed; later starts open no outbound connections at all.
|
|
#
|
|
# VISION. HALOGEN_VISION_TOWER=1 finds the sidecar beside the checkpoint. This
|
|
# is the fleet's only image input since the FLM NPU endpoint was retired on
|
|
# 2026-09-11; audio input has no replacement.
|
|
#
|
|
# CLIENT CONTRACT (see serverAppConfigs-docs/LLM/halogen-flash-server.md).
|
|
# There are no server-side sampler defaults: a request that sends no
|
|
# temperature decodes GREEDY. The per-request budget defaults to 8,192 tokens
|
|
# and REASONING COUNTS AGAINST IT, so a budget that runs out mid-thought
|
|
# returns finish_reason "length" with empty content and the partial reasoning
|
|
# in reasoning_content. The cap is 65,536; above it a request is a 400, never a
|
|
# silent truncation.
|
|
#
|
|
# IMAGE. The unit runs the localhost/-aliased tag: this host's podman quadlet
|
|
# path cannot resolve registry-prefixed refs from local storage at create
|
|
# time. deploy-remote.sh pulls the ghcr.io ref and re-tags it before the
|
|
# restart. Pull=never keeps boots off the registry.
|
|
[Unit]
|
|
Description=frank-large — halogen-flash-server, Qwen3.8-Flash-Next (w4b) on gfx1151
|
|
After=network-online.target
|
|
|
|
[Container]
|
|
ContainerName=frank-large
|
|
Image=localhost/peonist-ai/halogen-flash-server:0.5.8
|
|
# Published port stays 7770: Hermes, open-webui, the harness and the homepage
|
|
# probe all point here. 8730 (the engine) is deliberately not published.
|
|
PublishPort=7770:8731
|
|
AddDevice=/dev/kfd
|
|
AddDevice=/dev/dri
|
|
# Read-write: the first start downloads into it. Contains only weights.
|
|
Volume=/home/dank/models/halogen:/models
|
|
# Fetch the weights on first start only (absent checkpoint gates it).
|
|
Environment=HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next
|
|
# Fleet identity: the id at /v1/models, /health and every response body.
|
|
Environment=HALOGEN_MODEL_ID=frank-large
|
|
# Native context of the model. Matches what both clients declare.
|
|
Environment=HALOGEN_CTX=262144
|
|
# Trimmed 2026-09-13 from the 524,288 the 09-12 doubling deployed: buys
|
|
# ~3.4 GiB of co-tenant headroom, keeps one full 262k conversation resident
|
|
# beside a ~139k one, or three concurrent 131k chats.
|
|
Environment=HALOGEN_KV_POOL_POSITIONS=401408
|
|
# Concurrency cap. Each slot is 111.0 MiB of O(1) state; the KV is ONE
|
|
# 401408-position pool carved into per-request regions, so slots cap how many
|
|
# requests run at once while the pool binds how many positions exist. The
|
|
# slots still OVERSUBSCRIBE the pool (3 x 262144 > 401408), so admission is
|
|
# pool-bound -- one full-length conversation plus a ~139k second, or three
|
|
# at 131k.
|
|
# LOWERED 4 -> 3 on 2026-09-12: frees ~111 MiB on a host with 7.1 GiB left,
|
|
# and three users is the real fleet size. A fourth concurrent request now
|
|
# queues behind the three rather than contending.
|
|
Environment=HALOGEN_KV_SLOTS=3
|
|
# Sizing is budget arithmetic, all of it. ONE budget covers
|
|
# the KV pool, the prefill arena and the vision tower together:
|
|
#
|
|
# budget = 121.2 host - 67.7 resident weights - reserve
|
|
# need = KV (~28 KiB/position) + arena + vision (1.97 measured)
|
|
# arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
|
|
#
|
|
# The arena term is linear in max_tok, and it reproduces five live restarts
|
|
# AND upstream's three published "on the device" figures to within 0.1 GiB —
|
|
# note those published figures are pool+arena totals, not pool alone, which is
|
|
# the misreading that first made a doubling look affordable here. Every corner
|
|
# this box offered at a reserve of 24:
|
|
#
|
|
# 16384 -> need 28.6 vs 29.5 FITS, 0.9 spare (doubling, 8-9% prefill cost)
|
|
# 32768 -> need 37.0 vs 29.5 POOL CLAMPED to 262144, deploy still green
|
|
# 65536 -> need 46.5 vs 29.5 OOM, will not boot; unit still reads ACTIVE
|
|
#
|
|
# None of those three gave both features; reserve 15 made the doubled pool
|
|
# fit (38.5 vs 37.0) but at 7.1 GiB of host headroom. On 2026-09-13 the pool
|
|
# came down to 401,408 to restore co-tenant headroom: need ~33.4 vs 38.5.
|
|
# THE POOL, ARENA AND RESERVE ARE A MATCHED SET, NOT independent tunables --
|
|
# the fit check's printed ledger on each boot is the arbiter, not these
|
|
# comments.
|
|
Environment=HALOGEN_MAX_TOK=32768
|
|
# RAM left for the n-gram table the engine reads through the page cache.
|
|
# LOWERED 2026-09-12 from 24 (vendor default is 20) to buy the doubled pool at
|
|
# full prefill speed. The fit check's OWN requirement is the line to honour —
|
|
# read from a clamped boot, not modelled:
|
|
# "524288 positions need ~35.0 GiB (plus 1.5 of margin)" -> budget >= 36.5
|
|
# budget = 121.2 host - 67.7 weights - reserve
|
|
# reserve 16 -> 37.5 (1.0 spare); 15 -> 38.5 (2.0 spare, deployed). At the
|
|
# old 524,288 pool, 18 -> 35.5 CLAMPED it to 262144, so the ceiling there was
|
|
# 16; at 401,408 (need ~33.4) it loosens, but a higher reserve still buys
|
|
# nothing.
|
|
#
|
|
# AT THE DOUBLED POOL THE CEILING WAS 16, AND 16 BOUGHT NOTHING. Tested live at 524,288: 15 -> 16 changed the fit
|
|
# margin only. The server held 103.5 GiB and reported "7.1 GiB left for
|
|
# everything else" at BOTH settings, because this reserve feeds the fit-check
|
|
# budget and is not an allocation — actual footprint is weights + KV + arena +
|
|
# vision, none of which it touches. A higher reserve therefore cannot give the
|
|
# host memory back; it can only take the pool away. 15 keeps the larger margin.
|
|
#
|
|
# Measured at 15 before that test: pool reserved at 524288 with no clamp and no
|
|
# WARNING, and the server held 103.5 GiB. The ledger
|
|
# reads "host memory left for everything else: 310 contiguous 2 MiB blocks
|
|
# (7.1 GiB total)" — against 2081 blocks / 12.1 GiB at the 24-reserve
|
|
# baseline. All 28 co-tenant containers stayed up (~4.3 GiB working set).
|
|
#
|
|
# The price at 524,288 was that the engine volunteered this note on every boot:
|
|
# when it runs out, BOTH they and this server can stall for minutes at a
|
|
# time at full CPU with no disk activity and no output, which is not a
|
|
# crash and needs no restart.
|
|
# That is the failure mode to watch, and it is NOT the legible one predicted
|
|
# above: a minutes-long stall that looks like a hang is harder to diagnose
|
|
# than a slow prompt. The 09-13 trim to 401,408 was the first application of
|
|
# the remedy. If it happens again, cut further:
|
|
# HALOGEN_KV_POOL_POSITIONS=262144 (hands back another 3.6 GiB), or
|
|
# HALOGEN_MAX_TOK=16384 (hands back 8.4) with reserve then free to rise to 24.
|
|
Environment=HALOGEN_HOST_RESERVE_GIB=15
|
|
# Image input (the sidecar ships inside the weights repo).
|
|
Environment=HALOGEN_VISION_TOWER=1
|
|
# Verbose engine logging: the vendor's first-line evidence for load,
|
|
# allocation and admission questions. Shipping it on means an incident report
|
|
# already has the full startup account instead of needing a restart-into-debug.
|
|
# HALOGEN_DMALLOC_LOG is deliberately NOT set — it logs every allocation
|
|
# over 64 MB, which is steady-state noise here; raise it only for a live
|
|
# OOM hunt.
|
|
Environment=HALOGEN_VERBOSE=1
|
|
# Required by the image: the engine maps the checkpoint and registers it with
|
|
# the GPU in place, which needs host IPC and an unconfined seccomp profile.
|
|
PodmanArgs=--ipc=host --security-opt seccomp=unconfined
|
|
Ulimit=memlock=-1:-1
|
|
# The image ships its own HEALTHCHECK; this pins it. /health asks the ENGINE
|
|
# for a PONG before answering, so 200 means "ready to serve", not "process
|
|
# alive". During the first-start download it stays unhealthy by design.
|
|
HealthCmd=halogen-healthcheck api
|
|
HealthInterval=30s
|
|
HealthTimeout=10s
|
|
HealthRetries=3
|
|
HealthStartPeriod=300s
|
|
Pull=never
|
|
|
|
[Service]
|
|
# The first start reads ~68 GiB off disk and pins it before serving.
|
|
TimeoutStartSec=900
|
|
Restart=on-failure
|
|
RestartSec=10
|
|
|
|
[Install]
|
|
WantedBy=default.target
|